Why Context Beats Prompts

Every team that has spent real hours with an AI coding agent has watched a colleague author the perfect prompt — a paragraph of goal, three worked examples, an explicit list of what not to touch — and get back something adequate on Monday and something erratic on Tuesday, against the same repository, with what looks on screen like the identical careful instruction. The engineer’s first move is almost always to blame the instruction: tighten the wording, add another example, restate the constraint in bolder terms. On inspection, the instruction was rarely the problem.

Everything the prompt was sharing space with

By the time that instruction reaches the model, it is one entry in a much longer document the model actually reads: a system prompt tuned for a different task two releases ago, a tool definition the agent will never call this session, the full output of a grep run four turns back that nobody scrolled past, a file pasted in for context that has since been edited on disk without the session knowing. None of that is invisible to the model. It is competing, token for token, with the instruction the engineer spent twenty minutes wording — and that competition is exactly what a purely prompt-focused fix cannot touch.

Anthropic’s engineering team put a name to this in September 2025: context is “a critical but finite resource” for AI agents, and managing it is a different job than the one prompt engineering describes. Prompt engineering, as the field has documented it — a 2024 survey catalogued more than fifty distinct techniques — is about the construction of a single instruction or query: how to phrase it, which examples to attach, what reasoning structure to request of the model. Context engineering is about everything else that lands in the window alongside that instruction, accumulated across a session that may run for dozens of turns, not just the most recent one.

The distinction is easy to miss because the two disciplines look identical from the outside — both eventually produce a block of text the model conditions on. The difference shows up in what a fix touches. A prompt-engineering fix rewrites the instruction: sharper verbs, an added constraint, a worked example closer to the actual task. A context-engineering fix might not touch the instruction at all. It might drop a stale tool result from three turns back, defer loading a file until the step that actually needs it, or trim a system prompt written for a different kind of session. Two engineers debugging the same erratic output can walk away having changed completely different things, and only one of them is addressing what is actually crowding the window.

The unit of leverage moved

This is the reframe the rest of the volume assumes, so it is worth stating plainly: a model’s output is a function of everything present in its context at generation time, not only the most recent instruction. Two sessions can carry identical prompt text and diverge completely, because one inherited a window cluttered with stale search results and an oversized tool roster and the other did not. Treating the prompt as the unit of leverage — the thing to retune when results go sideways — was a reasonable model when interactions were mostly single-turn and the context surrounding the instruction was thin. It stops being reasonable the moment a session runs long enough, and touches enough tools, that most of the context window is not the instruction at all.

Context engineering, named

Andrej Karpathy gave the shift its name in mid-2025, describing context engineering as “the delicate art and science of filling the context window with just the right information for the next step.” The phrasing does real work in three places at once. “Delicate,” because getting it wrong in either direction — too little context or too much — degrades output. “Art and science,” because it resists being reduced to a checklist. And “just the right information,” because the goal is not maximization; a fuller window is not a better one. Anthropic’s formalization a few months later gave the coinage an engineering vocabulary to match: context is finite, additional tokens carry diminishing and eventually negative returns as they compete for the model’s attention, and the practitioner’s job is curation, not accumulation.

That last point cuts against a marketing instinct that has shaped how context windows get sold: bigger is framed as strictly better, a specification number worth chasing upward. The finite-resource framing says the opposite — a window twice the size does not double what a session can reliably hold, because the model’s attention over that window does not scale for free, and because a larger window that gets filled indiscriminately degrades in ways a smaller, curated one does not. Context engineering exists because the honest answer to “how much context should this session carry” is never “as much as fits.” It is “as much as earns its place, and no more.”

Two ways to fill the window

Anthropic’s practical taxonomy splits the discipline into two complementary strategies, and naming both is the closest thing this chapter offers to a checklist. Static context is loaded up front, before the model does any task-specific work: system prompts, tool definitions, few-shot examples, and any document the session’s designer decided the agent will need regardless of what the specific task turns out to be. Just-in-time context is fetched only when the work in progress calls for it — an agent holds a file path or an identifier and reads the file itself, at the moment it is relevant, rather than having every file it might touch preloaded into every session whether or not it ends up mattering. The pattern echoes how a competent engineer works an unfamiliar codebase: not by memorizing the whole repository before writing a line, but by looking things up as the work demands it, the pattern the literature calls progressive disclosure.

Neither strategy is categorically correct, and the choice is not binary in practice. A system prompt describing an agent’s role and hard constraints belongs in static context — fetching it just-in-time would mean the agent might act for several turns without knowing its own boundaries. A thousand-line file the agent might touch does not belong loaded by default into every session on the chance that one of them needs it. Most working harnesses mix both deliberately: a lean static core, plus tools the agent uses to pull in exactly what a given step needs, then lets that context age out once the step is done.

Coding agents make the trade-off concrete. A harness that reads a whole repository into context before the first tool call is choosing static loading for something that clearly belongs just-in-time — most of that repository is irrelevant to any single task, and every irrelevant file competes with the two or three that matter. A harness that instead gives the agent a directory listing and a file-read tool, and lets it request the specific files a task turns out to touch, is applying progressive disclosure to the same problem: the agent still ends up with everything it needs, but only after the session has established that it needs it, and the window never carries the files it does not.

Every irrelevant token is a debt taken early

There is an economic reading of this that the Library keeps returning to. Context Debt — Volume I’s term for the gap between context that once existed and context available now, paid down only when it is persisted somewhere durable — is usually described as accruing after the fact, once a session ends and nothing survives it. But a session can take on a version of that same debt preemptively, at the moment it loads context nobody needed: every token spent on a stale file, an unused tool definition, or an unread search result is a token unavailable for what the current step actually requires, and one that measurably competes with the tokens that do matter. Irrelevant context is not neutral padding. Per the context-rot evidence the next chapter treats in full, it can actively crowd out the context that would have produced a better answer — debt drawn down before the session even reaches its hard problem.

What this volume covers

This chapter names the shift and stops there deliberately — the mechanics get their own chapters. Context Rot, next, treats in full what this one only cited: why a context window’s advertised size is not its effective size, and why more tokens can make a session worse rather than better. Context Health, several chapters later, turns that diagnosis into a number a team can track per session instead of a feeling someone reports in standup. Further on, The Memory Ladder traces what happens when context stops being something a session merely fills and becomes something an agent also writes to — carrying forward across sessions instead of starting from zero each time. Each of those chapters assumes this one: that context, not the prompt, is the unit worth engineering.