An LLM context window is the working space a model can use for one request. It can hold your instructions, the current question, earlier conversation, documents or tool results you supply, and the answer the model generates. The limit is measured in tokens, not pages or words.
A larger context window lets a model accept more material. It does not guarantee that the model will remember every detail, find the right sentence, or keep using all of that information well. The useful skill is knowing what to include, what to leave out, and when to retrieve only the relevant pieces.
What an LLM context window actually is
Think of the context window as the information available to the model while it creates a response. It is sometimes compared to working memory: the model can refer to what is inside the current request or conversation state while answering.
This is different from the model’s training data. Training shapes patterns and information stored in the model’s parameters. The context window contains information supplied for the current task. A model can use a report you attach while that report is in context, but attaching it does not retrain the model or add it permanently to what it knows.
For the basic mechanics of how the model generates text from its current input, see our beginner guide to what an LLM is.
What counts toward the context window?
The context is more than the text you type into the chat box. Depending on the product and API, the request may include:
- System or developer instructions
- Your current message and earlier conversation turns
- Documents, images, or other supplied files
- Tool descriptions and results
- Structured formatting instructions or schemas
- The model’s generated output, including internal or reasoning tokens where applicable
Each provider has its own tokenization and accounting rules. A long PDF or image may use tokens even though you cannot count them by looking at the visible characters. API message boundaries and tool definitions can also add tokens that are not obvious in plain text. OpenAI’s token-counting guide explains how request structure, tools, and files affect input counts; Anthropic’s context-window guide describes what its API includes in a request.
The context window also has to leave room for output. If a request uses most of the available capacity for instructions, conversation, and source material, the model may have less room to answer. A separate maximum-output setting can impose a tighter ceiling still. The exact interaction depends on the model and endpoint, so check that provider’s current documentation rather than relying on a generic token-to-page estimate.
Context, model knowledge, and memory are different
These three ideas often get mixed together:
- Context: the input the model can reference for the current response, including supplied messages, documents, and tool results.
- Model knowledge: patterns encoded in the model’s parameters during training. It can be incomplete, outdated, or wrong.
- Persistent memory: information an application stores outside the current request and may retrieve for a later interaction.
If you tell a chatbot “I prefer concise replies,” the service might keep that preference in its memory feature, or it might only be available in the current conversation. That depends on the product. In an API integration, a request is often independent unless the application sends earlier messages again or uses a provider’s conversation-state feature.
Memory is not the same as a larger context window. A memory system stores or retrieves selected information; context is where the retrieved information is placed so the model can use it now. Our article on how AI agent memory works across sessions covers the storage-and-retrieval side.
Why can long context still fail?
A model can accept a long input and still miss a detail inside it. Three different problems are easy to confuse:
The request is too large
If the full request exceeds a model or endpoint’s context limit, the service may reject it or an application may need to shorten it. Chat products can manage an expanding conversation by dropping older material, summarizing it, or asking you to start a new thread. What happens is product-specific.
Important information gets diluted
A prompt may fit and still be crowded with irrelevant text. If a model must read a long archive to answer one narrow question, useful details compete with everything else. More material can make it harder to focus, and it can add processing time or usage cost depending on the service.
The model misses details in the middle
Researchers have studied whether a model can find a relevant fact based on where it appears in a long input. Liu and coauthors’ 2024 paper, “Lost in the Middle: How Language Models Use Long Contexts”, tested particular models on multi-document question answering and key-value retrieval. In those tests, the models often did better when the needed information appeared near the beginning or end than when it sat in the middle.
That paper is evidence of a real failure mode, not a promise that every current model behaves the same way on every task. But it is a good reason to test your own prompts. Anthropic’s current documentation also cautions that a larger context window does not make more context automatically better, and describes declining recall as the input grows.
Why a bigger window is not always the answer
A large window is useful when the task genuinely needs a lot of source material at once: comparing several contracts, reviewing a long codebase excerpt, or summarizing a long conversation. But maximum capacity and useful context are different things.
Sending everything can make a request harder to inspect and more expensive to process. It can also hide the few details the model needs. In many applications, selecting relevant sections or retrieving them on demand works better than pasting an entire knowledge base into every request.
This is where retrieval, chunking, and summarization help. Break a large source into useful sections, find the parts that match the question, and send those with enough surrounding context to make them understandable. Our guide to chunking and reranking explains how retrieval systems choose which passages to include. For long chats, context compaction covers how a system can summarize older turns when the working set grows.
How to decide what belongs in context
Before sending a large input, ask a few practical questions:
- What must the model know to answer? Include relevant instructions and evidence, not every related document you have.
- Can you retrieve a smaller set? Search, filter, or chunk a large collection and provide the best matching passages.
- Does the input contain repeated material? Remove duplicate history or summarize it, while keeping facts that affect the answer.
- Is there room for the response? Leave enough capacity for the expected output and check the model’s output limit.
- Can you test retrieval? Place a known fact near the beginning, middle, and end of a long document and ask the same question each time.
When accuracy matters, ask for source references or supporting quotes and verify them. If the model cannot point to the right passage, a bigger input may not fix the issue. A better retrieval strategy or a narrower question might.
FAQ
What is an LLM context window?
It is the maximum amount of tokenized information a model can use in a request, including supplied context and generated output. The exact limit depends on the model and endpoint.
Does the context window include the answer?
In common API designs, generated output counts against the model’s context capacity or is constrained by a separate output limit. Check the provider’s rules; some products expose these limits differently.
Is a context window the same as memory?
No. Context is available to the model now. Persistent memory is stored separately and may be retrieved into a later request. A product’s memory feature does not mean every past chat is loaded into every prompt.
Does a larger context window make an LLM smarter?
No. It gives the model room to receive more information. It does not add training knowledge or guarantee that the model will use every supplied detail correctly.
Why does an LLM forget something I included?
The detail may have been truncated, buried among irrelevant material, or hard for the model to retrieve from its position. Test with shorter, more focused context and ask for evidence from the source.
Try the position test
Write a short document with one fact you know. Put the fact near the beginning, then move it to the middle, then to the end. Keep the question identical and compare whether the model finds the fact each time. If you use a very long document, note where the answer becomes less reliable.
The point is not to chase the largest context number. Give the model the information it needs, in a form it can use, and test the result. Next in the roadmap series, we’ll look at LLM capabilities and the failure modes that still matter even when the prompt fits.
Featured illustration generated with AI.