Context window

What the model can see

A language model has no memory between requests. Every time you send a message, the app sends the whole conversation again, and the model reads all of it from the start. The amount of text it can read at once is its context window, measured in tokens.

When a chat grows longer than the window, something has to go. The simplest apps drop the oldest messages. The model then truly does not know what was said there, and it may contradict it or ask again.

Try it. Shrink the window and watch the start of the chat fall out of view. Then look at the last question.

How big the windows have become

Context windows grew fast. GPT-2 read about a thousand tokens in 2019, a few pages of text. In 2024 Gemini 1.5 Pro accepted a million, several thick books at once. By 2026 a million tokens is the usual size for flagship models, open ones included, and Llama 4 Scout is listed with ten million.

A token is about three quarters of an English word, so a page of 500 words is roughly 670 tokens. Other languages usually need more tokens for the same text, as the tokenization lesson shows.

Why long context is expensive

In attention every token looks at every token before it. Double the text and there are four times as many pairs to score. The square grows fast, so long prompts take more time per token, and some providers charge more for very long ones.

Memory grows too. The KV cache keeps the keys and values of every token in every layer, so a million tokens of context needs a lot of fast memory on the server.

Try it. Move the slider and compare how the pairs grow with how the text grows.

Lost in the middle

Fitting into the window does not mean being used well. A 2023 study called Lost in the Middle hid the needed fact at different places in a long prompt. The models found it best at the very start or the very end, and worst in the middle.

Newer models handle long text much better, but the advice still holds. Put the important instructions and the question at the start or the end, not buried in the middle of a long document.

Try it. Move the fact through the prompt. The curve shows the typical shape of the published results, not exact numbers.

Living with a limited window

Apps use a few tricks when a conversation gets too long. They drop the oldest turns, they replace them with a short summary written by the model, or they store the whole history outside and fetch only the relevant parts with RAG.

Each trick loses something. Dropping loses facts, a summary loses details, and a search can miss. Here the keyword search looks for the words of the last question, and nothing in the allergy message shares a word with a question about dessert. Long-running assistants usually combine the tricks and search by meaning.

Measured in

Tokens, about 0.75 English words each

Cost

Attention grows with the square of the length

Rule of thumb

Important text at the start or the end

Tricks

Drop old turns, summarize, retrieve