One word at a time

When you send a message to ChatGPT or Claude, the reply looks like a finished thought being typed out for show. It isn't. The model builds the reply one small piece at a time, and every piece is a fresh decision.

Here is the whole loop. The model reads everything so far, your message plus whatever it has already written. It gives a score to every possible next piece of text, tens of thousands of candidates. One gets picked and stuck on the end. Then the loop runs again with that piece included. A 300-word answer is a few hundred trips around the loop.

Scores become odds

The raw scores aren't probabilities yet. A step called softmax turns them into odds that add up to 100 percent. After "The cat sat on the", a model might give "mat" the best odds, "floor" and "sofa" a decent chance, and "moon" almost none.

Then the model's job is done. It doesn't choose. A few lines of ordinary code called the sampler roll dice weighted by those odds, and whatever comes up is the next word.

Temperature

Before the dice roll, the scores get divided by a setting called temperature. Below 1, the gaps between scores grow, so the favorite pulls further ahead. Above 1, the gaps shrink and long shots get a real chance. At 0, the sampler skips the dice and always takes the top choice.

Try it. The odds are made up, but the math is the real thing.

Slide the temperature to 0 and sample a few times. You get "mat" every time. Push it to 2 and "keyboard" starts turning up. Neither end is correct. Low temperatures suit jobs with one right answer, like pulling a date out of an email. Higher ones give variety, which helps a brainstorm and hurts a calculation. Most chat apps pick a setting for you and don't show it.

The dice explain two things you've probably noticed. Ask the same question twice and you can get two different answers. And a model can talk itself into a corner. It never goes back to change a word, so once an unlikely word is picked, everything after it has to make sense of it.

Small task, big consequences

"Predicting the next word" sounds trivial. It isn't. To predict the next line of a chemistry textbook well, it helps to have absorbed some chemistry. To continue a detective story, it helps to keep track of who was in the room. Whether that adds up to understanding is a real debate, and you'll meet it at the end of this course. The mechanism isn't up for debate. One piece at a time, picked by dice the model loaded.

One correction before moving on. I've been saying "word". Models don't actually work in words, and the next lesson shows what they use instead.