How LLM inference works is easier to understand if you picture a short loop, not a machine searching a database. The model reads the prompt, calculates what token should come next, adds that token to the running text, and repeats. The same loop runs whether you ask for a one-line answer or a long explanation.
This process is called inference: using a trained model to produce an output. It explains why an AI answer may take a moment to start, why text then streams in pieces, and why asking for a much longer response usually takes longer. We’ll trace one request from prompt to final token without getting lost in matrix math.
Inference is the model’s working phase
Training adjusts a model’s parameters using examples. Inference uses the trained parameters to answer a request. The model is not learning your prompt in the same way it learned from training data; it is processing the current input and producing a continuation.
That continuation is built from tokens. A token might be a whole short word, part of a longer word, punctuation, or another text fragment. The model turns the prompt into token IDs, works with those IDs internally, then maps generated IDs back into readable text. If you want a closer look at token boundaries, see our LLM tokenization guide.
For the common decoder-style text model, the central task is to estimate a distribution over possible next tokens. The model does not write a whole paragraph in one pass. It chooses one next token, appends it, and runs the next step with the expanded context.
How LLM inference works, step by step
1. The request becomes tokens
Your application sends the model an input: perhaps a system instruction, conversation history, and your latest message. An API may also include settings such as an output limit or a request to stream results. The provider converts the text into tokens that its model can process. The exact token count depends on that model’s tokenizer, not just the number of words.
2. Prefill processes the prompt
The model processes the input tokens to build an internal representation of the context. In a transformer, attention layers calculate how each position relates to earlier positions. This initial work is commonly called prefill. It prepares the model to generate and produces the scores used to choose the first output token.
A longer prompt generally gives prefill more material to process. That is one reason a large conversation history or a long document can affect the wait before the answer begins. The exact delay varies with the model, hardware, provider load, and serving setup; there is no universal “milliseconds per token” rule.
3. The model selects a token
For the next position, the model produces scores (often called logits) for tokens in its vocabulary. A decoding rule turns those scores into a choice. Some settings favor the highest-scoring option; others allow sampling among likely choices. Temperature and related controls affect that choice, but they are the next lesson in the series.
Once a token is selected, it is appended to the sequence. It could be a word fragment or even a space or punctuation mark, so the visible text may not arrive in neat word-sized chunks.
4. Decode repeats until a stop condition
The selected token becomes part of the context for the following step. The model predicts another token, appends it, and continues. This iterative phase is called decode. Generation ends when the model emits an end marker, the request reaches its output-token limit, or another supported stop condition is met.
That repeated loop is the practical answer to “How does an LLM write a response?” It keeps predicting a continuation from all the text available so far. Our earlier article, How LLMs Learn, explains how training creates the model that performs this task.
What the KV cache does
There is an obvious inefficiency in the simple loop above. If the model had to redo all attention calculations for the full prompt plus every earlier output token each time it generated one more token, it would repeat a lot of work.
Most transformer inference systems can keep a key-value cache, usually shortened to KV cache. Attention calculations create key and value information for tokens already processed. The cache stores that information so later decode steps can reuse it instead of recomputing the entire history from scratch. The current token still has to be processed, and the cache grows as the sequence gets longer.
Hugging Face’s Transformers documentation describes how cache entries are updated as new tokens are generated. It is a useful implementation-level reference, but the basic idea is simple: keep useful intermediate results from earlier tokens. Caching cuts repeated work; it does not make long context free. More context can mean more cache memory and more processing.
Some systems use different cache strategies or model architectures, so the exact memory use and speed vary. The key point for a beginner is not a formula. It is that the model can reuse information from the conversation while generating its answer.
Why the first token and the rest feel different
People often call every delay “latency,” but two parts of the experience are worth separating:
- Time to first token (TTFT): the wait between sending a request and seeing the first generated token. It includes request handling and prompt processing, among other provider-specific work.
- Generation speed: how quickly subsequent output tokens arrive once generation is underway. It may be described as tokens per second or as the gap between tokens.
The whole request also has total latency: from sending the request until the response is complete. A short answer can finish quickly even if its first token took a little while. A long answer can start promptly and still take time to finish because the model must run more decode steps.
These are useful measurements, not guarantees about what a person will perceive. Network time, queueing, model size, prompt length, hardware, batching, and provider choices can all change the result. When comparing services, match the model and workload, and check whether a reported tokens-per-second figure describes one request or a server handling many requests.
Streaming shows the loop as it happens
Without streaming, an API can wait until the model has produced the full answer and then return it as one response. With streaming, the application receives output events while generation is still running and can display text as it arrives. OpenAI’s Responses API guide documents streaming over server-sent events; other APIs have their own event formats and request options.
Streaming often makes a response feel more responsive because the reader sees the beginning before the model has finished. It does not mean the model generated the whole answer faster. It changes when your app can show the pieces. A developer also needs to handle partial output, completion events, and errors rather than assuming every stream ends cleanly.
Why longer answers usually take longer
Suppose you ask for a 50-token answer and then a 2,000-token explanation from the same model with the same prompt. The second request needs many more output-generation steps. Since ordinary autoregressive decoding chooses tokens in sequence, those extra steps usually add time.
That does not mean you can predict the exact duration by multiplying token count by a fixed number. Hardware, batching, output speed, reasoning behavior, and system load vary. Some models may also use internal reasoning tokens that count toward limits or work without appearing as ordinary visible text. The provider’s API documentation determines which token categories a particular limit includes.
Output length can affect usage charges too, but pricing rules differ by provider and model. Check the vendor’s current pricing page before estimating a bill; don’t assume one provider’s input/output rates apply elsewhere. A practical application should set a sensible output cap and ask for the amount of detail it needs, not “as much as possible.”
Hosted versus local inference
Inference can run on a provider’s servers or on hardware you manage. In both cases, the model still processes input and generates tokens. What changes is who operates the machines, chooses the serving stack, handles scaling, and manages access to data.
Hosted APIs are often simpler to start with. Local inference can give an operator more control, but it also brings hardware and maintenance decisions. We cover that operational choice separately in our guide to running OpenClaw with local models; it is not required to understand the token loop.
A quick walkthrough
Imagine you send: “Explain why leaves change color in autumn.” The application packages the prompt and sends it to the model. The model tokenizes the input, processes it during prefill, and scores candidate first tokens. It selects a token, then begins the decode loop. With streaming enabled, your screen may show “Leaves…” while the model is still generating the rest. Each new token extends the context; the cache helps reuse earlier attention data. When the model reaches a stop condition or output limit, the API marks the response complete.
The model did not find a finished answer card and paste it into the chat. It generated a sequence, one step at a time, using the prompt and its learned parameters.
Common misconceptions
- “The model thinks through the whole answer before showing it.” In standard autoregressive generation, output is produced token by token. Some products add other processing around that loop, but streaming can expose output before completion.
- “KV cache means the model remembers every past chat forever.” The cache is an inference-time aid for the current sequence. It is not the same thing as durable memory across separate conversations.
- “More tokens per second means every request feels faster.” A server-wide throughput number can hide the delay before your first token. Check TTFT and per-request generation behavior too.
- “A longer prompt only affects cost.” It can also require more prefill work and cache space, so it may affect response time and serving capacity.
FAQ
What does inference mean in AI?
Inference is running a trained model on an input to produce an output. For a text LLM, it commonly means processing a prompt and generating tokens in sequence.
What is the difference between prefill and decode?
Prefill processes the prompt and prepares the context for generation. Decode generates new tokens one at a time, reusing cached attention data when the serving system supports it.
What is a KV cache in an LLM?
It stores attention key and value data from already processed tokens so the model can reuse it during later generation steps rather than recomputing all earlier context.
Why does an LLM take time to answer?
The model must process the input before producing its first token, then run additional generation steps for the output. Network and server conditions add more variation.
Does streaming make the model generate faster?
Not by itself. Streaming lets an application display pieces of the response as they arrive instead of waiting for the entire answer.
Try it yourself
Send the same prompt twice: once asking for a one-sentence answer and once asking for a detailed explanation. If the interface streams text, notice the time before the first token and the pace afterward. The output may vary, but the exercise makes prefill, decode, and answer length less abstract.
Next in the roadmap series, we’ll look at sampling and generation parameters, including temperature and top-p. For now, remember the useful mental model: process the prompt, predict one token, append it, and repeat until the request stops.
Featured illustration generated with AI.