Context Windows Explained: Tokens, Limits and Costs

A model has no memory of your conversation. On every turn it is handed a block of text and continues it. The context window is the size limit on that block, and everything the model appears to know about your task, your documents and what was said five minutes ago has to fit inside it.

Most confusing behaviour in LLM applications, such as forgetting an instruction, ignoring a document or a bill that grows faster than expected, comes back to this one limit.

What it is

Anthropic's documentation defines the context window as all the text a model can reference when generating a response, including the response itself, and describes it as a working memory, separate from the data the model was trained on (Anthropic: context windows).

Two phrases there matter. "Including the response" means the reply shares the space with the prompt. OpenAI's documentation puts the same rule plainly: prompt and output together must not exceed the model's maximum context length (OpenAI: key concepts). And "working memory" means that nothing in it is learned. When the call ends, it is gone.

What counts against it

The second row surprises people. A chat feels continuous, but the API is stateless: the application sends the whole conversation again with each message. A long chat is a long prompt.

How big

Sizes are quoted in tokens, and they change with each generation of models. As read on 2 October 2026, Anthropic's documentation lists a 1M-token window for its current main models, with up to 128k tokens of output, and 200k for others. At about three quarters of a word per token, that is in the region of 750,000 and 150,000 words.

Local models are a different scale. Ollama sets the context length by available graphics memory, and on a machine with less than 24 GiB it defaults to 4k tokens, about 3,000 words (Ollama: context length). It can be raised, at the cost of memory.

Check the page for the exact model you use. The number is a property of the model and sometimes of your plan, not of "LLMs".

What happens at the limit

The script puts one invented fact at the start of a prompt, follows it with the first 1,700 words of PEP 8 (PEP 8) and asks for the fact at the end. It runs the same prompt on a small local model with three context lengths, then moves the fact to the middle and to the end.

import json
import urllib.request


def ask(prompt, num_ctx):
    """One call to a local model through Ollama, with the context length set for this call."""
    body = {"model": "llama3.2:3b", "prompt": prompt, "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_predict": 20, "num_ctx": num_ctx}}
    request = urllib.request.Request("http://localhost:11434/api/generate", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    reply = json.load(urllib.request.urlopen(request, timeout=1800))
    return reply["prompt_eval_count"], reply["response"].strip().split("\n")[0][:58]


sections = json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
pep8 = " ".join(s["text"] for s in sections if s["pep"] == 8).split()
document = " ".join(pep8[:1700])                      # the first 1,700 words of PEP 8
half = " ".join(pep8[:850]), " ".join(pep8[850:1700])
FACT = "Note for staff: the access code for the Hyderabad lab is 4417."
QUESTION = "Question: What is the access code for the Hyderabad lab? Reply with the number only."

print("the fact is at the START of the prompt; the context length changes")
print("context length   prompt tokens the model saw   answer")
for num_ctx in (1024, 2048, 4096):
    seen, answer = ask(f"{FACT}\n\n{document}\n\n{QUESTION}", num_ctx)
    print(f"{num_ctx:>14}   {seen:>27}   {answer}")

print()
print("the context length is 4096; the position of the fact changes")
for position, prompt in (("middle", f"{half[0]}\n\n{FACT}\n\n{half[1]}\n\n{QUESTION}"),
                         ("end", f"{document}\n\n{FACT}\n\n{QUESTION}")):
    seen, answer = ask(prompt, 4096)
    print(f"{position:>14}   {seen:>27}   {answer}")

the fact is at the START of the prompt; the context length changes
context length   prompt tokens the model saw   answer
          1024                          1024   I can't provide information about accessing a lab, includi
          2048                          2048   I can't provide information about accessing a lab, includi
          4096                          2444   4417

the context length is 4096; the position of the fact changes
        middle                          2446   4417
           end                          2444   The answer is 4417.

• The prompt is about 2,444 tokens. With a context length of 4096 the model saw all of it and answered 4417.
• At 1024 and 2048 the model saw exactly that many tokens and no more. The start of the prompt, where the fact was, had been cut off.
• Nothing reported a problem. There was no error and no warning. The model answered confidently that it could not help, which was true for the text it was given.
• Position made no difference here. With enough room, the fact was found at the start, in the middle and at the end.

Two cautions about reading this. It is one small model and a short prompt. And hosted services behave differently at the limit: Anthropic's API, for example, rejects a request whose input alone exceeds the window with an error saying the prompt is too long. Whether your stack fails loudly or cuts silently is something to find out before your users do.

On position, research with much longer inputs found a real effect. Models did best when the relevant information was at the beginning or the end, and noticeably worse when it sat in the middle (Liu et al.: Lost in the Middle).

Bigger is not automatically better

A window of a million tokens invites a simple plan: put everything in. Anthropic's own documentation advises against relying on that. As the number of tokens grows, accuracy and recall degrade, which it calls context rot, and it describes choosing what goes into the context as just as important as how much space there is.

There are two further costs:

• Money. Input is billed per token on every call. A document left in the conversation is paid for again on each turn.
• Time. More tokens take longer to process.

The arithmetic of a chat makes the first point concrete. Suppose each turn adds 500 tokens. The tenth call sends 5,000, the twentieth 10,000. Over twenty turns the application has sent 105,000 tokens of input to exchange 10,000 tokens of actual conversation. See cost optimization for LLM applications.

Managing it

1. Retrieve, do not stuff. Find the few passages that bear on the question and send only those. This is the idea behind RAG.
2. Compact old turns. Replace early parts of a long conversation with a summary. Both Anthropic and OpenAI now offer this on the server side (OpenAI: compaction).
3. Trim tool output. A tool that returns a 5,000-row table has just filled the window. Return what the model needs.
4. Put what matters at the edges. Instructions and the question belong at the start or the end, not buried in the middle of a long document.
5. Leave room for the reply.
6. Count tokens before sending. See tokenization in LLMs.
7. Start a new conversation for a new task. Old context is not neutral. It costs money and competes for the model's attention.

Common mistakes

• Assuming the model remembers. It sees what is sent, each time.
• Not knowing your limit. A local runner's default can be far smaller than the model's maximum.
• Trusting silence. No error does not mean the whole prompt was read.
• Pasting whole documents when a section would answer the question.
• Letting conversations run for days.
• Confusing the window with training. Putting a document in the context does not teach the model anything it will keep.

For how the limit follows from the way models compare tokens with each other, see attention explained.

The Generative AI course covers context windows and token management alongside RAG and embeddings.

Sources

• Context windows (Claude Platform Docs) — Anthropic
• Key concepts (OpenAI API documentation) — OpenAI
• Context length (Ollama documentation) — Ollama
• Lost in the Middle: How Language Models Use Long Contexts (arXiv) — Liu et al.
• Compaction (OpenAI API documentation) — OpenAI
• PEP 8: Style Guide for Python Code (Python Enhancement Proposals) — Python Software Foundation