Evals as an Engineering Practice

A platform team spent two weeks tightening the system prompt for the coding agent wired into their internal tooling. In the retro, everyone agreed the new version felt sharper — fewer over-eager rewrites, tighter diffs, less time spent re-explaining scope to the agent mid-session. They shipped it on a Friday. Ten days later, someone noticed the agent had quietly stopped running the project’s test suite before marking a task complete. Three merged pull requests carried regressions nobody caught until a customer did.

Nobody on the team had been careless. They had done exactly what a diligent team does: read transcripts, watched sessions, formed a shared sense that the change was an improvement. What they had not done was build anything that could tell them, mechanically, whether the new prompt still did everything the old one did. The gap between “this feels better” and “this is measurably not worse” is the gap this chapter is about.

Flying blind by consensus

Anthropic’s own account of how agent teams evolve names this pattern precisely. Early on, a combination of manual testing, dogfooding, and intuition is enough — the team is small, the changes are visible, and everyone who touches the product also uses it. That approach breaks down at a specific, recognizable moment: users report the agent feels worse after a change, and the team has no way to verify except to guess and check. Absent evals, debugging becomes reactive — wait for complaints, reproduce by hand, fix the bug, and hope nothing else regressed in the meantime.

This Library has already documented why “it feels better” is a dangerous instrument to trust, in a different but structurally identical setting. In METR’s randomized controlled trial, experienced developers using early-2025 AI tools were 19% slower on their own repositories while estimating they had been roughly 20% faster — a forty-point gap between measured and perceived outcome. That was not a one-time artifact: the same research group’s early-2026 follow-up found developers again overestimating a real speedup, in the opposite direction, with an honest caveat attached — developers increasingly refuse to work without the tools at all, which skews who is left to ask. The instrument under test there was a developer’s felt sense of their own throughput. The instrument in this chapter’s opening story is an engineer’s felt sense of an agent’s quality after a prompt change. It is the same instrument, and the evidence that it reads wrong does not stop applying just because the question changed from “am I faster” to “is this agent better.”

What an eval actually is

An eval, in Anthropic’s own framing, is a test for an AI system: give it an input, apply grading logic to the output, and use the result to measure success. That definition looks almost identical to a unit test’s assert-and-report shape, and the resemblance is intentional — an eval is software testing’s discipline, carried over to a system whose outputs are not deterministic. The difference that matters shows up after the first case. A unit test either passes or fails, once, and the answer is stable on rerun. An agent given the same input twice can behave differently both times, so a single run tells an engineer almost nothing about reliability. An eval suite runs the same or similar tasks across many trials and reports a rate — the number worth watching is not “did it pass” but “how often does it pass, and has that number moved.”

For agents specifically, the grading step is harder than for a single model reply, because the thing being graded is not one output but a trajectory — a sequence of tool calls, file edits, and state changes that can succeed by an unexpected path or fail while looking superficially fine. Some of that grading is deterministic: did the test suite pass, did the expected file get written, did the diff match a reference. Some of it requires another model to read the trajectory and judge it against a rubric, a practice researchers call LLM-as-judge — shown to approach human-level agreement on open-ended comparisons, and shown, in the same research, to carry its own biases toward verbosity and toward whichever answer it sees first. Grading logic is itself software, with its own bugs, a fact this chapter returns to before the end.

Capability evals and regression evals

Teams that get past the flying-blind moment tend to converge on two distinct suites, doing two different jobs. A capability eval asks whether the agent can now do something it could not do before — it is how a team decides an investment in a longer context window, a new tool, or a better-specified prompt actually paid off, and the target is a rising number with no fixed ceiling. A regression eval asks a narrower, less glamorous question: does the agent still handle everything it used to handle? A regression suite should sit near a hundred percent pass rate almost all the time, and a drop is the signal, not the level itself — it means something in the harness, the prompt, or the underlying model broke a capability the team was already relying on.

The workflow-level version of this practice is what separates a mature harness from a collection of clever prompts. Instead of testing one model call in isolation, a workflow regression suite runs the actual multi-step task a team’s agents perform every day — open a ticket, read the relevant files, make the change, run the tests, open the pull request — end to end, through the real harness, not a simplified stand-in for it. Anthropic’s own starting guidance is modest by design: twenty to fifty simple tasks drawn from real failures the team has already seen is enough to begin, not the hundreds a mature program eventually accumulates. The suite exists to answer one question on every change to the harness, the prompt library, or the model version: did this just break something that used to work?

Evals-in-the-loop

Software engineering solved a version of this problem two decades ago, and the solution has a name worth borrowing directly. Continuous integration, as originally described, depends on a codebase being self-testing — a comprehensive suite run before every integration, on every commit, specifically to catch regressions while they are still cheap to fix. Nobody today argues that running the test suite occasionally, or only when a change “feels risky,” is an adequate substitute for running it on every commit. The same argument applies to agent workflows, and Anthropic states it almost as a direct translation of the CI idea: automated evals are especially useful pre-launch and in CI/CD, running on each agent change and model upgrade as the first line of defense against quality problems.

Evals-in-the-loop means treating a prompt edit, a tool-schema change, or a model-version bump exactly the way a software team treats a pull request — nothing merges without the suite running, and a regression-suite failure blocks the change the same way a broken build does. The discipline is not exotic; it is continuous integration, applied to a system whose logic lives partly in weights instead of entirely in source code. What is still uncommon is how few teams running production agents have actually built it. Most of the industry’s evaluation effort goes toward capability leaderboards that track whether frontier models are improving in general — not toward the unglamorous regression suite that would catch a team’s own prompt edit quietly breaking its own workflow.

The eval that lies

Building the suite is not the finish line, because a grading function is code, and code has bugs — an eval can be wrong in either direction. Anthropic’s own account includes a case worth sitting with: Opus 4.5 initially scored 42% on CORE-Bench, a result that read like a capability gap, until a researcher traced the failures and found the grading itself was rejecting correct work — penalizing “96.12” for not matching an expected “96.124991…”, ambiguous task specifications, and stochastic tasks nobody could reproduce exactly. After the grading bugs were fixed and a less constrained scaffold was used, the same model scored 95% on the same benchmark. Nothing about the model had changed in the interim. The eval had been lying, and the number alone gave no hint of which direction the lie ran.

The corrective discipline is unglamorous and easy to skip under deadline pressure: read the transcripts. An eval score nobody has read behind is exactly the vibes problem this chapter opened with, one layer down — a figure that feels authoritative because it is a number, standing in for a judgment nobody actually checked. The practice worth copying is investing in tooling for viewing eval transcripts and reading them regularly, specifically because a failed task’s transcript is the only way to tell whether the agent made a genuine mistake or the grader rejected a valid answer. A regression suite a team stops reading is a regression suite the team is trusting blind — the exact failure mode it was built to eliminate in the first place.

What to do on Monday

1. Write down the twenty to fifty tasks your agent workflow already handles today in production, drawn from real work rather than hypothetical hard cases. That is a regression suite’s starting inventory.
2. Wire the suite to run on every change to the system prompt, tool definitions, or underlying model — triggered by the change itself, the same trigger a test suite uses for a commit, not by a calendar.
3. Set the regression pass bar near 100% and treat any drop as a stop-ship signal, not a metric to average away over the following week.
4. Keep the regression suite separate from any capability suite tracking whether the agent can now do more. They answer different questions and belong on different dashboards.
5. Budget time to read failing transcripts before trusting a score in either direction. A grading bug and a genuine regression look identical on a dashboard and completely different in a transcript.

None of this requires exotic tooling. It requires the same decision software engineering made when it stopped trusting “it worked when I ran it” as a release criterion: that a suite of representative cases, run automatically on every change, catches more than any individual’s judgment does — including the judgment of the person who wrote the change. Generation from an agent is cheap and getting cheaper. Knowing, mechanically, whether what it generated is what the team actually wanted is the discipline this volume is about, and evals are where that discipline starts.