Learning from a mountain of text

A brand-new model knows nothing. Its internal numbers, called parameters or weights, are set at random, and a large model has billions of them. Ask it to continue "The cat sat on the" and it produces gibberish. Everything it later knows comes from one simple exercise, repeated an enormous number of times.

The exercise

Take a passage of real text. At every position, have the model guess the next token. Compare its odds with what actually came next. Then adjust every parameter a tiny amount in the direction that would have made the right answer a bit more likely.

The adjusting uses a method called gradient descent. For each parameter, calculus works out whether nudging it up or down would have reduced the error, and roughly by how much. All of them get nudged at once. One round of nudges does almost nothing. Repeat it across trillions of tokens and the nudges add up to a model that can write.

Nobody has to label the data. The text is its own answer key, because the next word is always right there. That's why this stage can use so much of it.

Press Play or scrub the update step. The toy curve shows prediction error falling as training adjusts the weights.

What goes in

The training text for a large model is assembled from web pages, books, code, scientific papers, forums, and more. It's filtered to throw out spam, duplicates, and junk. For the largest models it runs to trillions of tokens. Training on it keeps thousands of specialized chips busy for weeks or months, which is why only a handful of organizations train the biggest models from scratch.

The mix matters a lot. More code in the mix makes a model better at code, and there's some evidence it helps with careful step-by-step reasoning more generally. Text that appears many times over tends to get memorized word for word. Deciding what goes in is one of the least glamorous parts of building a model and one of the most important.

What comes out

The result is called a base model. It does one thing extremely well. It continues text in a way that fits. Give it the start of a news story and it writes a plausible rest. Give it half a recipe and it finishes the recipe.

To get good at that, it ends up storing a great deal along the way. Facts, grammar in many languages, how arguments are usually built, what a polite email sounds like. None of it sits in a list you could read off. It's spread across the weights, and it's compressed. A model can't hold its training text word for word, because the weights are far smaller than the text. It holds something more like the patterns, with common things stored sharply and rare things stored as a blur. Keep that blur in mind. It explains a lot in the lesson on why models make things up.

Bigger has kept helping

Researchers found that when you scale up the model and the data together, with the computing power to match, prediction error falls along a smooth, predictable curve. These patterns are called scaling laws, and they're the main reason models grew so quickly. How long the curve keeps paying off, and whether lower error keeps turning into new abilities, is an open argument.