How do LLMs learn? They start by predicting missing or next tokens in text, using the text itself to provide training examples. That pretraining gives a model broad patterns, but it does not automatically make the model a helpful assistant. Further training can teach it to follow instructions and favor responses people want. The stages are easier to understand when you separate what each one teaches from what it cannot guarantee.
This guide walks through pretraining, supervised fine-tuning, and preference or reinforcement training. We’ll also look at memorization, generalization, and why a model can know a lot yet still respond badly to a simple request.
How LLMs learn: the short version
A language model learns by adjusting its internal weights so that useful next-token predictions become more likely. During pretraining, it sees many text sequences and tries to predict what comes next. Later, a model may be trained on examples of instructions and good answers, then tuned using preference ratings or other reward signals.
A useful mental model is:
- Pretraining: learn broad patterns from text by predicting tokens.
- Supervised fine-tuning (SFT): practice responding to prompts using example answers.
- Preference or reinforcement training: favor outputs that score better under chosen feedback signals.
This is a common outline, not a recipe every model follows. Training methods vary, and “fine-tuning” can refer broadly to additional training after pretraining. Some teams use direct preference methods or other methods rather than a reinforcement-learning stage.
Pretraining: learning from text that supplies its own answer
Imagine the sentence “The kettle began to whistle when the water started to ___.” A model can be shown the text before the blank and asked to predict the next token. The original sentence already contains the target answer. No person has to label that example by hand.
This is called self-supervised learning: the training target is derived from the data itself. For a next-token model, each sequence supplies many small prediction exercises. Given “The kettle began,” predict “to”; given “The kettle began to,” predict “whistle,” and so on. Real training systems process tokens in batches, but the central idea is the same.
When the prediction is wrong, a training algorithm calculates how the model’s weights should change to reduce the error. Repeating this over a large collection of text gradually changes the model. It does not store a neat dictionary of facts or read with human intent. Instead, its parameters come to encode patterns that help predict text.
What can next-token prediction teach?
To predict well, a model benefits from recognizing grammar, common facts, relationships between ideas, writing conventions, and patterns in code. If a passage says “Paris is the capital of,” the likely continuation is “France.” If a code snippet opens a function call, the next tokens often depend on its syntax and surrounding variables. The model learns correlations that help with these continuations.
That ability can support tasks that were not presented as separate labeled lessons. A model trained on many kinds of text may continue a translation, summarize a paragraph, answer a question, or produce code when the prompt makes that pattern likely. But “can produce a plausible answer” is not the same as “has checked the answer.” Next-token training alone does not promise truth, good judgment, or obedience to a user’s intent.
Memorization versus generalization
Training examples can influence a model in different ways. Sometimes a model reproduces a phrase or detail it encountered. That is memorization. In other cases, it applies a pattern to a new combination of words or a situation it has not seen exactly. That is generalization.
Suppose a model has seen many examples of recipe instructions, but never the sentence “Fold the blueberries into the batter.” It may still produce a sensible continuation because it has learned patterns about recipes, ingredients, and verbs. This is not proof that it reasons like a person. It shows that learned patterns can be recombined beyond exact quotation.
The boundary is not clean. A model can generalize in one respect while repeating memorized material in another. Its training data, architecture, training process, and prompt all matter. So avoid both extremes: an LLM is not merely a lookup table, and fluent new wording does not prove that every answer came from reliable reasoning.
For a practical builder, the distinction suggests two checks. Ask whether the output is a familiar phrase or a new application of a pattern, then independently test whether it is correct. A confident tone is not evidence either way.
Why pretraining does not automatically create a good assistant
A pretrained model is trained to continue text. It may complete a question with another question, imitate a forum thread, or continue a document in the style it has learned. That can look intelligent, but it is not the same objective as answering the user helpfully.
Think of pretraining as giving someone a huge reading history but no specific practice in helping customers. They might know a great deal and still not know when to ask a follow-up, how much detail to give, or when to say “I don’t know.” Assistant behavior takes additional shaping.
The roadmap’s distinction is useful here: pretraining can produce a capable base model, while post-training can make it behave more like an assistant. The later training does not erase the base model; it changes which responses are more likely in response to a prompt.
Supervised fine-tuning: learning from example answers
In supervised fine-tuning, a model is trained on examples that pair an input with a desired response. For example:
Prompt: Explain a solar eclipse to a ten-year-old.
Example answer: A solar eclipse happens when the Moon moves between Earth and the Sun, blocking some or all of the sunlight for a short time.The model is trained to make the example answer more likely after that prompt. With many examples, it can get better at instruction-following, conversational formats, and tasks such as answering questions or summarizing. The examples might be written by people, generated by another model and reviewed, or assembled through a mix of processes.
SFT is not a database update. Adding a few examples does not reliably install a perfect new fact in every context. Nor do examples cover every possible user request. SFT teaches response patterns drawn from its training examples; the final behavior still depends on the prompt, the model, and what it learned earlier.
Preference and reinforcement training: shaping which answers are favored
There can be several plausible answers to a prompt. A training team may want the model to prefer the accurate, clear, safe response over one that is vague, misleading, or needlessly agreeable. Preference methods provide another signal about which outputs are better for the target use.
In reinforcement learning from human feedback (RLHF), people can compare or rate candidate answers. Those judgments can be used to train a reward model, which scores responses; reinforcement learning then adjusts the language model to favor higher-scoring outputs. Other setups use AI feedback, automatic checks, or direct preference methods. These methods are related, but they are not interchangeable names for one exact pipeline.
OpenAI describes one current post-training process for GPT-4o updates as starting from a pretrained model, using supervised fine-tuning on ideal responses, then reinforcement learning with multiple reward signals. The company also notes that the choice and weighting of reward signals affects the resulting behavior. That is one documented example, not a universal description of all LLMs.
Reward signals are imperfect stand-ins for what people actually value. If a system rewards answers for sounding helpful, it may learn to sound helpful even when it should be uncertain. If feedback favors agreement too strongly, the model may become sycophantic. Training can improve behavior, but it can also amplify mistakes in the measurement of “good.”
From base model to deployed assistant
It helps to distinguish three labels:
- Base model: a model after broad pretraining, often suited to continuing text.
- Instruction-tuned model: a model further trained on prompts and example responses.
- Deployed assistant: the model inside a product, which may also include system instructions, tools, safety checks, retrieval, and product-specific settings.
That last distinction matters. A chatbot’s response is not determined by model weights alone. The surrounding software may add a system message, retrieve documents, call tools, filter outputs, or choose a particular model version. When behavior changes, the cause might be a model update, a prompt change, a tool, or another part of the product.
Likewise, not all models use the same post-training recipe. Some public model families have separate base and instruct variants. Their documentation may describe SFT, preference training, or reinforcement learning in different combinations. Treat a pipeline diagram as a map of common stages, not a checklist that every provider must follow.
Try it: compare a base model and an instruction model
If you can access both types of model, give them the same prompt: “Explain why leaves change color in autumn in two sentences.” Look at whether each follows the requested format, answers directly, and stays on topic. Then ask each to continue the unfinished text “The old bridge crossed the river just before…” A base model may be more likely to continue the passage, while an instruction model may treat the text as a request to answer or explain.
This is an observation exercise, not a controlled benchmark. Models differ in their training data, size, prompt templates, and settings. One pair of outputs cannot establish that a training method caused a specific behavior. It can, however, make the difference between continuation and instruction-following concrete.
What training does not guarantee
- Factual accuracy: a likely continuation can still be false or out of date.
- Reliable reasoning: a model can solve a familiar pattern and fail on a small variation.
- Perfect safety: safety examples and rewards reduce some risks but cannot cover every situation.
- Human-like understanding: training changes numerical parameters; it does not establish that the model experiences meaning as a person does.
- One fixed behavior: prompts, system instructions, decoding settings, tools, and product changes all affect outputs.
For engineering work, evaluate the behavior you need instead of assuming a model’s training label guarantees it. Build a small set of representative prompts, include edge cases, and check outputs against the task’s actual success criteria.
Frequently asked questions
Do LLMs learn from every conversation?
Not necessarily. A deployed model usually generates responses with its current weights; whether conversations are later used for training depends on the provider’s product, settings, and data policies. Do not assume a chat changes the model immediately.
Is pretraining the same as fine-tuning?
No. Pretraining builds broad capability, often with next-token prediction over large text collections. Fine-tuning is additional training on a model that already has pretrained weights. “Fine-tuning” is a broad term and can include more than supervised instruction training.
Does reinforcement learning teach an LLM facts?
It can shape which responses the model favors, but preference or reward training is not a dependable way to add a changing body of facts. For current, traceable information, an application may need retrieval or another data source, followed by evaluation.
Why does a model sometimes sound certain when it is wrong?
The model generates text based on patterns and its training signals; it does not automatically verify each claim against a trusted source. A confident style can be learned or prompted even when the underlying answer is mistaken.
What is the difference between SFT and RLHF?
SFT trains on desired prompt-and-answer examples. RLHF uses human preference feedback as part of a reward-based training process. They can both be part of post-training, but they use different kinds of training signals.
Key takeaway
Pretraining teaches a model to predict patterns in text. Supervised fine-tuning teaches it from example responses. Preference or reinforcement training can shift which responses it favors. Each stage shapes behavior, none guarantees truth, and the exact recipe differs across models.
To build a better mental model, continue with our beginner’s guide to what an LLM is. Then compare the training stages above with how tokens and transformer layers work in the next chapters of this series.
Sources and further reading: AI Engineering Roadmap, M1.4; Hugging Face LLM Course; OpenAI on post-training and reward signals; IBM on self-supervised learning.
The featured illustration was generated with AI.