Judge Panels & Fleet Verification

A team running one agent against one ticket eventually produces a diff a human reads, the way engineers have read diffs for thirty years. A team running several agents against several variants of the same ticket — to compare approaches, or simply to get more attempts at a hard problem — produces something a human review process was never built to consume: a stack of correct-looking pull requests, each internally consistent, each passing its own tests, most of which should not ship. The bottleneck has moved from writing the code to deciding which of the several plausible versions of the code to keep.

This is a different problem than code review, not a bigger version of it. Code review assumes one artifact and asks whether it is good enough to merge. Fleet judging assumes several artifacts of unknown relative quality and asks which one is best, fast enough that the answer arrives before the next batch finishes. The two questions look similar and are not: a diff can be individually mergeable and still be the worst of five options, and a team that reads every candidate the old way — full context, full attention, one at a time — pays a review tax that scales with the fleet rather than the ticket.

What an LLM judge actually buys

The established term for the fix is LLM-as-judge: using a model, rather than a person, to score or compare the outputs other models produced. The technique is not new. Zheng and colleagues formalized it in 2023, showing that a strong judge model agreed with human preference judgments in roughly 80% of cases — about the same rate at which two human judges agree with each other. That number is worth sitting with: it means an LLM judge is not a degraded substitute for human review so much as a different reviewer of comparable, imperfect reliability, useful precisely because it is available at a volume no human reviewer is.

How elaborate that judge needs to be is where the genuinely counterintuitive result lives. Anthropic’s account of building its multi-agent research system describes testing the more obvious design against a simpler one. The obvious design used multiple judges evaluating each output from different angles. The simpler one used a single LLM call, given one rubric, returning a 0.0-to-1.0 score and a pass/fail grade. The single call won: Anthropic reports it was “the most consistent and aligned with human judgements” of the approaches tried, and it let them evaluate outputs by the hundreds rather than by the dozen.

The evidence does not all point one way

Treat that as a data point, not a law. A separate line of research reaches something closer to the opposite conclusion in a different setting: Verga and colleagues’ Panel of LLM Evaluators found that a jury of several smaller, differently-sourced models outscored a single large judge on open-ended generation tasks, with less bias from any one model’s quirks, at roughly a seventh of the cost of repeatedly running one large judge. The two findings are not strictly contradictory — Anthropic’s task was scored against a fixed, mostly objective rubric; the panel study covered open-ended generations with no single correct answer — and neither result has been independently replicated at the other’s scale. The honest summary is that the field has not settled whether simplicity or plurality wins the judging question in general. What is verified is narrower and still useful: for agentic tasks graded against a concrete rubric, a single well-specified scoring call has proven a defensible, cheap starting point, not a universally superior one.

Cheap and simple is also gameable. Hwang and colleagues showed in 2025 that persuasive but wrong answers — responses padded with confident, consistency-signaling language rather than correct content — inflate LLM judge scores by several percentage points on average, and that the bias survives switching to a larger judge model or adding an explicit counter-prompt warning it not to be fooled. A passive scorer can be talked into a good grade. That is the case for the second technique this chapter covers.

Adversarial verification: judging by trying to break it

A passive judge asks: on a scale of zero to one, how good is this? An adversarial verifier asks a different question: does this actually work, and what happens when I try to break it? The distinction sounds small and is not. Passive scoring reads the artifact and forms an opinion; adversarial verification interrogates it — runs the test the author didn’t write, feeds it the input it wasn’t designed for, argues the counter-case before accepting the case. It costs more per judgment, because an adversarial pass usually means another round of tool calls rather than one inference, and it catches a different class of failure: not “is this well-written” but “is this actually true,” which is precisely the gap a persuasive, wrong answer exploits. The two techniques are complementary rather than competing — a cheap passive score for triage across a large fleet, an adversarial pass reserved for the small number of finalists an actual decision rides on.

Planner, worker, judge

None of this is useful without somewhere to plug it in. This volume named the orchestrator-worker pattern in Chapter 4: one process decomposing a goal into parallel tasks, dispatching them to workers, and assembling the results. Anthropic’s own taxonomy of agent workflows separately describes an evaluator-optimizer loop, in which one model generates and another critiques in a cycle. The planner/worker/judge hierarchy this chapter’s evidence keeps arriving at is what you get by composing the two: a planner decomposing the goal, workers executing in parallel, unchanged, and a judge — passive or adversarial, single-call or paneled — evaluating the results before anything is accepted, rather than after a human notices something wrong in production. The judge is the addition. Everything else is Chapter 4’s pattern, run at a scale where a human can no longer be the implicit judge standing at the end of it.

Cursor’s fleet, both ways

The most publicly documented existence proof for judged fleets at real scale is Cursor’s own account, published in January 2026, of running hundreds of concurrent coding agents against a single codebase. In one run, the target was a web browser, FastRender, built from nothing; the agents worked for close to a week and produced over a million lines of code across roughly 1,000 files. In a second, more mundane run against Cursor’s own product, the target was a live React migration that ran for over three weeks and touched on the order of 266,000 added and 193,000 removed lines. Cursor’s architecture matches the hierarchy above closely: planners that decompose work and can spawn sub-planners recursively, workers that pick up tasks and push changes without coordinating with each other, and — at the end of every cycle — a judge agent deciding whether to continue. The company’s own retrospective is candid about the failure modes along the way: without a judge or hierarchy at all, agents held locks too long or never released them, and left unsupervised they turned risk-averse, avoiding hard problems in favor of small, safe changes that were individually defensible and collectively useless.

The reception to the browser demo supplied the other half of the lesson, and it arrived fast. Independent commentators who examined the actual repository found rendering glitches and styling breaks consistent with genuinely novel code rather than a wrapped browser engine — evidence the agents had built something real, not staged a demo. Critics went further: reporting on the project’s GitHub Actions history cited job failure rates around 88%, read as evidence of a codebase that mostly did not run, and at least one former browser-engine contributor called the result bloated relative to hand-written engines of comparable size. Cursor’s own post hedges the flashier result — the browser reads as a research artifact, not a shippable one — while noting that even the more disciplined React migration, the one that passed CI, “still needs careful review.” Read together, the two examples make a point this volume keeps returning to in different forms: a planner/worker/judge fleet can produce a genuinely large amount of working-ish code, fast, and producing a large amount of code fast is not obviously the goal. The review burden a fleet creates does not vanish because a judge triaged it first — it moves, the same way review time moved when generation got cheap in Volume I’s accounting of where AI-assisted work’s costs actually land — and someone still has to decide whether a million lines nobody has fully read is an asset or a liability.

A framework for choosing a judge

A judge that can silently become the decision was never really advisory.