Context Rot

A team upgrades to a model with a much larger advertised context window — 200K tokens where the old one offered 32K — and treats the upgrade as license. Whole files go into the prompt instead of relevant excerpts. Entire design docs ride along for background. Full conversation histories get reattached on every follow-up, on the theory that more information can only help, and a bigger window finally means there is room for all of it. The team expects sharper answers. What it gets instead is a model that misses a detail sitting in the middle of a document it was just handed, contradicts an instruction from three screens up, or produces an answer that would have been correct for a shorter, more curated version of the same prompt. Nobody can reliably reproduce the miss. The prompt is roughly the same shape as the one that worked cleanly yesterday. The failure looks random.

It is not random, and the model has not failed to read the text — it processed every token, in the same technical sense that a camera captures a wall of text lit brightly at one end and shadowed at the other. What changes is how reliably the model can act on what it read, and that reliability depends on how full the window is, what else is competing for attention inside it, and where in the input a given detail happens to sit. Volume I’s chapter on Context Debt borrowed a name for this in passing, as the mechanism behind lost decisions and re-explained architecture: context rot. This chapter treats it as a phenomenon in its own right — what the evidence actually shows degrades, why it degrades unevenly, and why the instinct to paste in more is not the free move it feels like.

What the benchmarks actually show

Chroma Research’s 2025 evaluation ran eighteen models — spanning the GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 families — through three kinds of tasks built to separate “can the model find information” from “does it use that information reliably as the surrounding context grows.” Extended Needle-in-a-Haystack variants tested not just lexical retrieval but semantic matching, the effect of distractors, and how similar the needle was to the haystack around it. LongMemEval ran 306 conversational-QA prompts averaging roughly 113,000 tokens, comparing performance on a focused context against the full one. A Repeated Words task asked models to reproduce a sequence exactly, with one unique word inserted, across 1,090 variations from 25 to 10,000 words. Across all three, the paper’s central claim held: “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows” — on tasks simple enough that length should not have mattered at all.

The specific findings are what make the claim more than a restatement of “long prompts are harder.” A single distractor measurably reduced accuracy, and the size of that reduction was not uniform — some distractors did far more damage than others, and the gap widened as input length grew. Degradation accelerated when the needed fact was phrased differently from the question asking for it, meaning a model’s ability to use a fact depends on more than the fact being present. And in the study’s most counterintuitive result, shuffling a haystack into incoherent order — destroying its structure — improved performance relative to a cleanly organized one, consistently, across all eighteen models. Structure that looks helpful to a human reader is not guaranteed to be helpful to a model managing a finite attention budget.

The pattern that predates it

None of this is a new observation arriving with new vocabulary. In 2023, Liu and colleagues studied how language models handle long contexts using multi-document question answering and key-value retrieval, and found a consistent U-shaped performance curve: models were reliable at retrieving information from the start and end of a context and unreliable in the middle, a primacy-and-recency bias that held across model families and persisted even in models explicitly built for long-context use. The finding predates the 2025 Chroma work by two years and several model generations, and it foreshadows it almost exactly — a context window was never a uniform bucket that fills evenly, and the bias toward the edges over the middle showed up the first time anyone measured carefully for it.

Put the two findings together and the shape of the problem clarifies into two separable axes. Where a detail sits in the window matters, independent of anything else — the “lost in the middle” effect. And what else is competing for the model’s attention at that position matters too — distractor count, semantic similarity between the fact and the question, even the coherence of the surrounding structure. Both reduce to the same underlying claim: not every token inside a model’s context window is equally “visible” to it, and visibility is shaped by more than how many tokens fit.

Capacity is not reliability

This is the distinction worth making explicit, because vendor context-window numbers invite a category error. An advertised window — 32K, 200K, a million tokens — is a capacity claim: a statement about what fits. It is not a reliability claim about what the model will use well once it is in there. NVIDIA’s RULER benchmark made the gap concrete: evaluating seventeen long-context models across thirteen tasks more demanding than the simplest retrieval test, it found that only about half could sustain satisfactory performance at 32,000 tokens — a length well inside what most of those models advertised as supported — even though nearly all of them scored close to perfect on the easy, single-needle version of the same test that vendors tend to quote.

That gap between the number on the spec sheet and the number that actually holds up under harder evaluation is exactly what context rot names. A model’s effective context — the portion of the window it uses reliably — is smaller than its advertised context, and it is not smaller by a fixed discount that can be estimated once and reused. It shrinks unevenly, by position, by what is competing nearby, and by how directly a needed fact is phrased relative to what is being asked of it. A prompt that fits inside the window is not the same claim as a prompt the model will read the way the task needs it to.

A context window’s advertised size answers whether information fits. Context rot answers whether the model will actually use it — and those are different questions with different answers.

On the gap between capacity and reliability

Why “just paste more in” is not free

The practical consequence is that adding context has a cost even when it comfortably fits, and the cost is not intuitive from the outside. Every additional file, thread, or paragraph pasted into a prompt is a potential distractor competing for the same finite attention the task’s actually-relevant details need, and Chroma’s finding that even one distractor moves the needle means that cost accrues from the first irrelevant addition, not just once the window is nearly full. Organizing the extra material carefully does not reliably buy the discount it seems like it should — the shuffled-haystack result says a model’s use of structure can run opposite to a human reader’s intuition about what “well organized” looks like. The lesson is not that curation is impossible to get right; it is that curation cannot be replaced by volume, however tidy the volume is packaged.

What a context budget actually buys

None of the four rows argues for a smaller window or a hard rule about token counts. They argue that the number worth tracking is not how much fits, but how much of what is in the window is actually earning its place — which is a curation question, answered by what goes in and where, not a capacity question answered by the vendor’s spec sheet.

What to check this against

Operon does not yet have a published, statistically powered distribution linking prompt composition to reliability, and this chapter will not manufacture one. What follows is the shape of the check a team can run against its own sessions, not a result — illustrative figures, offered as a starting instrument.

Where the rest of this volume goes from here

Anthropic’s framing of context engineering treats the window as a finite, curated resource rather than a bottomless pipe, and that framing is the practical answer to everything this chapter has documented: the question is never how much can be crammed in, but which configuration — what is loaded up front, what is retrieved just-in-time, what is left out entirely — is most likely to keep the model’s effective context aligned with its advertised one. Volume I’s Context Debt named what happens when context is lost between sessions and has to be re-earned; context rot is the different, earlier mechanism, describing what happens to reliability inside a single session’s window before anything is lost at all. The rest of this volume builds the response to both, piece by piece: a way to score a session’s context health in something more concrete than a feeling, a contract for what a harness’s compaction keeps and drops when a session runs long, and a ladder from static retrieval up through context an agent actively curates and maintains for itself.