Tokens, not words

A model never sees letters. Before your message reaches it, a program called a tokenizer chops the text into chunks called tokens and swaps each chunk for an ID number from a fixed list. The model only ever works with those numbers.

Where the chunks come from

The list, called the vocabulary, is built before training by counting text. Start with single characters. Find the pair that shows up side by side most often and glue it into a new chunk. Repeat that many thousands of times. Common words like "the" end up as one token, and rare words get built from pieces. A typical vocabulary holds tens of thousands of tokens, sometimes a few hundred thousand. The method is called byte-pair encoding, and most models use some version of it.

Watch the spaces. In English, a token usually carries the space in front of it, so " cat" with a space and "cat" without one are different tokens. "Cat" with a capital is a third.

Pick a sentence below. The splits are illustrative. Every model has its own tokenizer, but they all behave roughly like this.

Tap Show what the model sees. That row of numbers is the whole input. Everything the model does starts from there.

Odd behavior that tokens explain

• Spelling. Asked how many r's are in "strawberry", models have often stumbled. The model gets something like "str", "aw", "berry" as three IDs. The letters are hidden inside the chunks, and the model has to have learned how each chunk is spelled. It usually has, but not reliably.
• Arithmetic. Long numbers get split into chunks that don't line up with ones, tens, and hundreds, so 48213 might arrive as 482 and 13. Working digit by digit gets awkward. Newer tokenizers split numbers more regularly, and a model that can call a calculator avoids the problem.
• Cost and limits. Prices and the limit on how much a model can read at once are counted in tokens. In English a token is roughly three quarters of a word, so 1,000 tokens is about 750 words.
• Other languages. Tokenizers built mostly from English text break other languages into more pieces. The same sentence can take two or three times as many tokens in some languages, which means slower, pricier answers and less room to work with.

Why not just use letters?

Letters would make every text several times longer, and the model's main operation gets expensive fast as text grows. Whole words fail the other way. No fixed list can hold every name, typo, and newly coined word. Tokens are the compromise. They're long enough to keep texts short, and since the vocabulary includes every single byte, any text at all can be encoded, even emoji and words that didn't exist last year. Unusual text just costs more tokens.

So when you type a message, the model receives a short list of numbers. The next question is how a number like 9246 can mean anything at all.