Your 1M-token context window is a lie (context rot)
The pitch for million-token context windows is that you can stop doing retrieval and just throw everything at the model — the whole codebase, the whole ticket history, the whole manual. I've done it. It feels powerful. Then the model confidently misses a detail that was sitting right there in token 400,000, and you realise the number on the box and the amount the model can actually use are two very different things.
The study that put numbers on it
Chroma's "Context Rot" research (July 2025) tested 18 models — the Claude 4 family, GPT-4.1 and friends, Gemini 2.5, Qwen3 — and found that performance is not uniform across context length. Even on deliberately simple tasks, accuracy falls as the input grows, and it doesn't fall smoothly: models hit cliffs well before the advertised limit. The drop can be substantial — tens of points — long before you fill the window. (I'm quoting the shape of their finding, not a tidy "X%" number; the exact figures that float around in summaries aren't a Chroma quote, so I won't dress one up as one.)
The mechanism is the interesting part. Degradation isn't driven by token count alone. It's driven by:
- How semantically similar the distractors are to your question. Irrelevant-but-related text hurts more than obviously-unrelated text. A haystack full of near-misses is worse than one full of noise.
- Where the relevant information sits.
- The structure of the haystack — and here's the genuinely counterintuitive bit: across all 18 models, accuracy was higher on a shuffled haystack than on a logically coherent one. Our intuition that a well-organised long document is easier for the model is just wrong.
This isn't new, it's just measured now
Two older results back it up. NVIDIA's RULER benchmark introduced the phrase "effective context length" — the point past which a model stops holding up — and found that of a batch of models all claiming 32K or more, only a handful actually stayed useful at 32K. GPT-4 claimed 128K; its effective context was closer to 64K. And the classic "Lost in the Middle" paper showed the U-shaped curve: models use information at the start and end of the context far better than anything buried in the middle. Recency and primacy, in a language model.
"Effective context" is to the advertised window what a car's real-world range is to the number on the brochure. Both are true. Only one of them gets you home.
What I actually do about it
The takeaways are unglamorous and they work:
- Retrieval plus a tight, relevant context still beats dumping everything in. Effective context is much shorter than nominal, so a smaller, curated payload is both more reliable and cheaper. The "just use the big window" instinct is the lazy 2026 anti-pattern.
- Prune distractors, don't just maximise recall. Because semantically-similar noise is the worst kind, the quality of what you leave out matters as much as what you put in. Ten near-miss chunks are worse than three good ones.
- Put the important stuff at the edges. Lead with it or end with it; don't bury the one line that matters in the middle of a 50K-token dump.
- Measure on your own task. Effective context is model- and task-specific. Before you trust a 200K window for something that matters, run a small eval at the lengths you actually use.
The fair caveat
Long context isn't a lie in the sense of "useless" — it's a lie in the sense of "much smaller than the sticker says." Big windows genuinely help when you can't predict what to retrieve, for whole-document reasoning, for multi-turn agents, for cutting RAG plumbing you'd otherwise maintain. And models differ: the spread in RULER was real, and newer frontier models degrade less than the ones from a year ago. The honest claim isn't "long context doesn't work." It's "effective context is far shorter than advertised, and it degrades unevenly — so curate like it's scarce, because for the model, it is."
Which, pleasingly, is the same lesson as writing for humans: don't make your reader wade through everything to find the one thing that matters. The model is a worse reader than you think, and it never admits it.