# Meaning as a direction

> Each token becomes a point in a huge space, where nearby points and shared directions carry meaning.

Lesson 3 of 9 in [How LLMs actually work](https://learnfold.app/learn/llms/), a free course from Learnfold.

A token ID is just a label. Token 9246 isn't any closer in meaning to token 9247 than to token 12. So the first thing a model does is look up each ID in a big table and swap it for a long list of numbers, often a few thousand of them. That list is called an embedding.

### A list of numbers is a point

Two numbers pin a point on a map, how far east and how far north. Three numbers pin a point in a room. A list of a few thousand numbers pins a point in a space with a few thousand directions. You can't picture that, but the arithmetic works the same way. Two points can be close together or far apart, and you can draw an arrow from one to the other.

Nobody chooses these numbers. They start out random and get adjusted during training, nudged in whatever way improves the model's predictions. What falls out is that tokens used in similar ways end up near each other. "Cat" lands near "dog" and "kitten". "Tuesday" lands near "Wednesday". The model was never told that cats and dogs are related. Putting them close together helped it predict text, so it happened.



Drag the map sideways or use the slider. Related words keep their neighbors as the illustrative space turns.

### Directions carry meaning too

The surprising part is that directions can mean something, as well as distances. In a famous result from earlier word models, the arrow from "man" to "woman" pointed roughly the same way as the arrow from "king" to "queen". Start at king, subtract man, add woman, and you land near queen. Similar arrows turned up for country to capital and for present tense to past.

Hold that story loosely. The famous examples were picked because they worked, and plenty of analogies come out messier. The broad idea has held up, though, and it goes further inside modern models. Researchers keep finding directions that seem to track things like what language a text is in, whether a statement is true, or whether the writing is formal.

### Meaning shifts with context

The lookup table gives "bank" one starting point, whether you mean the side of a river or a place that holds money. That can't be the end of it. As a token passes through the model's layers, its list of numbers is rewritten again and again using the words around it. By the middle layers, "bank" in "we fished from the bank" and "bank" in "the bank denied my loan" have moved to very different places.

So when people say a model represents meaning, this is roughly what they mean. A word's meaning, to a model, is a position and a set of directions in a huge space, learned because they helped predict text. Whether that deserves the word "meaning" in the full human sense is a fair question. It is clearly much more than counting which words appear together.

The rewriting step, where words reshape each other, is the heart of the whole machine. That's the next lesson.

### Rabbit holes

> Why would putting similar words close together help a model predict the next word?

> How do researchers find a direction inside a model that tracks something like truth?

> Do two different models end up with similar maps of meaning?

Read online: https://learnfold.app/learn/llms/embeddings/