Benchmark Skepticism

A platform team picking a coding agent for a company-wide rollout does what any reasonable team would do: it opens three vendor pages and compares the number each one leads with. One reports 74% on some benchmark, another 70%, a third — Claude Code, in one widely cited cross-harness study — 88.6% on SWE-bench Verified. The team picks the highest number, writes it into the rollout memo as the justification, and moves on. Nobody in the room has run SWE-bench themselves, read one of its problem instances, or asked what fraction of their own codebase’s actual bug reports resemble anything in it.

That instinct is not unreasonable. SWE-bench is the most cited benchmark in AI coding precisely because it looks like the real thing: 2,294 problems pulled directly from GitHub issues and their accepted pull requests, across twelve popular Python repositories, introduced in an October 2023 paper by Carlos Jimenez and coauthors at Princeton. An agent has to read an actual bug report, edit an actual codebase, and pass the actual regression tests the maintainers wrote — a harder and more realistic task than the synthetic coding quizzes it displaced. The best model in the original paper, Claude 2, solved 1.96% of it. Three years later, vendors report scores above 70%. The number moved a great deal. What the number means moved less than the marketing implies.

What the score is a score of

SWE-bench itself changed twice, both times because the original version had problems worth fixing. SWE-bench Lite, a 300-instance subset released in March 2024, existed to make evaluation cheap and fast — but its filtering criteria selected for single-file fixes with short problem statements, which rewards agents that are good at narrow patches and says little about the cross-file reasoning the full benchmark was built to test. SWE-bench Verified, released by OpenAI in August 2024 working with the benchmark’s original team, existed for a more serious reason: human review found that a meaningful share of the original problems had test cases too narrow or too broad to fairly grade a correct fix, problem statements ambiguous enough to admit multiple valid solutions, or environment setups that made tests fail regardless of the patch. What survived that human filter became the 500-instance Verified set — the version nearly every current leaderboard number, including the 88.6% figure above, is quoting. A score from the 2023 original, the Lite subset, and Verified are three different measurements. A citation that does not say which one it used is already withholding information.

The illusion

In June 2025, a Microsoft Research team — Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam — published a study with a title precise enough to double as its finding: “The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason.” Their method was simple and hard to argue with. They asked frontier models to name which file in a repository contained a bug, using nothing but the issue description — no repository access, no tool calls, pure recall. On SWE-bench problems, models got this right up to 76% of the time. On structurally similar localization tasks drawn from repositories outside SWE-bench, the same models’ accuracy on the same kind of question fell to 53%. A second test — asking models to reproduce the exact buggy function from memory — showed the same shape: up to 35% verbatim five-word-sequence overlap with the ground truth on SWE-bench Verified and the full benchmark, against 18% on comparable out-of-benchmark tasks. Both results point the same direction. The authors call the pattern repository-bias memorization — the models are not uniformly skilled at localizing and recalling code, they are unusually skilled at it specifically on the repositories and issues that ended up in this one benchmark, consistent with those issues, or close paraphrases of them, having been present during training.

Passing is not the same as correct

A second, independent line of critique attacks the grading rather than the training data. SWE-bench marks a patch “resolved” if it passes the test suite the original pull request shipped with — and that suite is not always strong enough to tell a correct fix from a merely convenient one. A 2025 study led by You Wang, Michael Pradel, and Zhongxin Liu built a differential-testing tool that ran each “resolved” patch against additional developer-written tests beyond the ones SWE-bench checks. 7.8% of patches counted as resolved failed those extra tests outright. Among patches that passed everything the benchmark requires, 29.6% still produced runtime behavior different from the reference fix the maintainers actually merged, and manual review judged 28.6% of those divergent patches — roughly one in twelve of all “solved” instances — certainly incorrect. OpenAI ran a comparable audit from a different angle before it announced, in February 2026, that it would stop reporting SWE-bench Verified scores at all: reviewing 138 problems its own o3 model had consistently failed, it found 59.4% contained flawed tests — some enforcing implementation details the issue never specified, others checking for functionality the issue never asked for. The same audit found every frontier model it tested, across three labs, could reproduce verbatim gold patches or issue-specific details on certain Verified tasks — contamination, not reasoning. Two teams, two methods, converging on the same shape of result: a nontrivial share of what the benchmark calls solved is either not actually solved, or not fairly graded, or both.

What a maintainer would say

The most direct test of whether a passing patch is good code is asking the people who would have to accept it. In a March 2026 study, METR recruited four maintainers across three of the repositories SWE-bench draws from — scikit-learn, Sphinx, and pytest — and had them review 296 AI-generated patches the way they review any other contribution, blind to whether the automated grader had marked it passing. The gap was large: maintainer merge decisions ran about 24 percentage points below what SWE-bench’s own scoring reported. Even the benchmark’s human-written reference solutions — the ground-truth patches the entire grading system is built around — were merged only 68% of the time on fresh maintainer review, a reminder that some of the gap is intrinsic noise in what “correct” means rather than a defect specific to AI output. The AI-specific gap was still stark: for Claude Sonnet 4.5, patches passed the automated grader roughly 80% of the time and were judged mergeable by a maintainer only about 50% of the time. Rejections clustered around code-quality problems, changes that broke other functionality, and fixes that did not address what the maintainer understood the issue to actually be — exactly the judgment a pass/fail test suite cannot structurally capture.

Eighty percent passed the grader. Fifty percent would have been merged. The other thirty points were never in the score.

On METR’s 296-patch maintainer-review study

Reading a benchmark number

Not an economics input

None of this makes SWE-bench worthless. Forcing the field to grade agents against real GitHub issues instead of synthetic snippets was a genuine advance, and even a contaminated, imperfectly graded number correlates loosely with real capability at the frontier. The point is narrower and more actionable: a SWE-bench percentage is a benchmark score, not a business metric, and treating it as the latter skips every step this Library’s economics volume insists on. It says nothing about the mix of tasks a specific team actually runs, which rarely resembles a curated set of Python issues from a dozen popular open-source repositories. It says nothing about a specific codebase’s conventions, its own test suite’s strength, or its review culture’s tolerance for near-misses. It says nothing about cost — a model that solves five more points of SWE-bench at triple the token spend is not a win by the only measure that pays anyone’s bill. What predicts value for a specific team is what Session ROI and Cost Per Outcome were built to measure in the first place: did sessions run on this tool, on this codebase, on this task mix, produce changes that merged and stayed merged, at a cost the team can defend. A vendor’s leaderboard rank is, at best, a reason to run that trial. It is not evidence the trial is unnecessary.

The discipline this chapter is arguing for is not a one-time correction. Three months after OpenAI recommended SWE-bench Pro as the harder, less contaminated replacement for Verified, an internal audit of that same replacement reportedly found roughly thirty percent of its public tasks broken in ways that made scores unreliable, because — like the benchmark it replaced — its problems were scraped from real pull requests rather than purpose-built and independently verified. Benchmark skepticism is not a verdict to reach once about SWE-bench and file away. It is a standing habit to apply to whatever benchmark replaces it next.

1. Before quoting a SWE-bench figure, name the variant. Original, Lite, and Verified are different tests; a vendor page that cites the number without the variant is choosing the most favorable one.
2. Treat any single-benchmark percentage as a screening filter, not a decision. It tells you which tools are worth trialing, not which tool will perform for your team.
3. Run the trial on your own repositories, with your own task mix, and grade it the way METR graded maintainer review — did it merge and stay merged — not the way an automated test-suite grader would.
4. Track cost alongside capability. A benchmark score has no denominator; Cost Per Outcome does.
5. When the next benchmark replaces this one, apply the same audit before adopting its number — contamination and weak grading are properties of how benchmarks get built from real-world data, not defects unique to SWE-bench.