Why it makes things up

Ask a model for court cases that support some legal argument, and it may hand you case names, courts, and years that look perfect and don't exist. This is called hallucination. It sounds like a strange glitch, but it follows directly from everything so far.

Two kinds of memory

A model has two places to get information from.

The first is its weights. Everything it absorbed in pretraining lives there, compressed. Things that appeared thousands of times in the training text, like the boiling point of water, are stored sharply. Things that appeared a handful of times, like the third author of an obscure paper, are smeared. The weights also stop at a cutoff date. The model knows nothing about events after its training text was collected, unless something tells it.

The second is the context window, the text the model can see right now. Your conversation, the system prompt, a document you pasted in. This is exact. If the answer is written in the context, the model can usually quote it correctly.

The window is measured in tokens. Early models could see a few thousand. Some can now take hundreds of thousands, enough for several books. Anything outside the window is invisible. A model doesn't remember yesterday's chat. When an app seems to remember you, it's saving notes and pasting them into the context of your next conversation. Even inside a big window, models can be less reliable about material buried in the middle of a long text than about its beginning and end.

How blur becomes confident fiction

When the model needs a fact that isn't in the context, it falls back on the weights. For a common fact that works fine. For a rare one, the weights hold the general shape: what a case citation looks like, which names sound like judges, which years are plausible. The model was trained to produce text that fits, and a made-up citation fits perfectly.

Nothing in the next-token loop checks facts. The model writes a fake case name with the same fluent machinery it uses to write "Paris". There's some evidence that models carry an internal signal of how sure they are, but that signal doesn't reliably reach the words. Training can make things worse, too. If graders reward a confident answer over "I don't know", the model learns to guess.

Sample several guesses, then add the fictional archive entry. A plausible name becomes checkable only when you compare it with evidence.

What helps

• Put the source in the context. Paste the document, the email thread, the table. The model then works from exact text instead of the blur. Search features do this for you, as the next lesson shows.
• Ask for quotes. "Quote the sentence that supports this" is much easier to check than a summary.
• Distrust specifics. Names, numbers, citations, links, and quotations are where fabrications hide. A general explanation of a well-known topic is usually safer than a precise detail about an obscure one.
• Make room for not knowing. Telling the model that "I'm not sure" is an acceptable answer can cut down on guessing. It won't stop it.

Hallucination has become less common as models improved, but nobody knows how to make it zero in a system built on producing plausible text. Treat a model like a well-read colleague who never says "let me check". Often right, and worth checking on anything that matters.