From text predictor to assistant

A base model is a strange thing to talk to. Type "What is the capital of France?" and it might reply "What is the capital of Germany? What is the capital of Spain?" On the web, a question like that often sits in a list of quiz questions, and the model is continuing the document. It isn't answering you. Turning it into an assistant takes more training.

First, show it examples

People write example conversations, often many thousands of them. A user asks something, and an assistant answers well. The model trains on these exactly as before, predicting the next token, but now every text has the shape of a good conversation. This is called supervised fine-tuning. It's a tiny amount of data next to pretraining, and it works because the base model already knows how to write. It mostly needs to learn which kind of document it's in.

Behind the scenes, a chat is still one long text. Your messages and the model's replies are stitched together with special marker tokens that say who is speaking. Most apps also put a system prompt at the top, instructions you usually don't see that set the tone and the rules. The model continues that document in the assistant's voice and stops when it produces a special end-of-turn token.

Then, learn from preferences

Writing perfect answers by hand is slow. Judging which of two answers is better is much faster. So in the next stage, the model writes several answers to the same prompt and people rank them. Those rankings train a separate reward model, which learns to predict what people will prefer. Then the assistant is trained with reinforcement learning to produce answers the reward model scores highly.

This is called reinforcement learning from human feedback, or RLHF. Some variants replace part of the human judging with an AI judge that follows a written set of principles. Others skip the separate reward model. The aim stays the same. Steer the model toward answers people actually want, and away from harmful ones.

Tap the training stages to compare the responses. The wording changes as training steers the model toward answering helpfully.

For some models, practice with checkable answers

Some newer models get another round of reinforcement learning on problems where the answer can be checked, like math problems with known solutions and code with tests. The model writes out long working before its final answer and gets rewarded when the answer is right. These are often called reasoning models. That written working is useful, but it isn't a perfect window into how the model got there. The last lesson comes back to this.

Side effects

Preference training shapes more than helpfulness. People tend to rate confident, agreeable answers highly, so models can drift toward flattery and toward telling you what you want to hear. Researchers call this sycophancy, and labs work to counter it. Refusals also come from this stage. Some are well judged and some are overcautious.

Here's the useful split to remember. Most of what a model knows comes from pretraining. Its manners, its format, and its willingness to say "I'm not sure" come mostly from this later tuning. That's why two assistants built on similar base models can feel like very different characters.