The Productivity Paradox

Every AI rollout eventually produces a slide. Somewhere in a quarterly review, an engineering leader stands up and says a version of: the team feels significantly faster since we adopted the new tooling. The slide behind them usually carries a survey number — seventy percent, eighty percent, whatever the internal pulse check returned — next to a green arrow. Nobody in the room asks what the arrow is measuring, because everybody in the room already believes the underlying claim. The team says it is faster. The team, surely, should know.

That belief turns out to be one of the least reliable readings available in software engineering, and not because engineers are dishonest. Feeling faster is a real sensation, generated by real working conditions, and those conditions changed the moment code started appearing on screen faster than anyone could type it. The best available evidence says the feeling and the fact can point in opposite directions at the same time, and the gap between them is wide enough to invalidate almost any staffing or tooling decision made on sentiment alone.

The experiment nobody expected

Most published studies of AI coding tools measure something narrower than they claim to. A model solves a self-contained benchmark problem, or a group of developers builds a small feature they have never seen before, in a repository they do not maintain. Those studies are not wrong, but they answer a different question than the one engineering leaders are actually asking, which is: does AI make senior engineers faster at the real, ambiguous, half-specified work they already do every day, in the codebases they already know cold?

In 2025, METR ran the first rigorous attempt to answer that exact question. Sixteen experienced open-source developers worked in their own repositories — mature projects averaging more than twenty-two thousand GitHub stars and over a million lines of code, places where the developers already carried years of accumulated context. They supplied 246 real issues from their own backlogs: bug fixes, features, and refactors that were going to happen regardless of the study, work they judged genuinely valuable to their own projects, each one taking about two hours on average. Each issue, not each developer, was randomly assigned to allow AI tools or forbid them, which meant the same person did comparable work in both conditions and could not simply be a stronger or weaker programmer in one arm than the other.

The result: tasks where AI was allowed took nineteen percent longer to complete, not shorter. Before the study began, the developers forecast that AI would cut their completion time by twenty-four percent. After finishing the work — after living through the slower, AI-assisted version of it — they still estimated that AI had saved them about twenty percent of their time. The forecast was wrong by roughly forty-three points in one direction; the lived-experience estimate, formed after the work was already done, was wrong by nearly the same amount in the other. Two separate chances to notice the slowdown, and the same instrument missed it both times.

They predicted AI would save nearly a quarter of their time. It cost nineteen percent instead, and having just finished the work, they still believed it had saved them a fifth of it.

The full forecast-to-actual swing in METR’s early-2025 randomized trial

Why the design choice matters

The design is what makes the result worth taking seriously, and it is also what makes the result easy to misread as a verdict on AI coding tools in general. It is not that. In 2023, a separate randomized trial gave roughly ninety-five developers a single, bounded, well-specified task — implement an HTTP server in JavaScript, from scratch, as fast as possible — and split them into a group with GitHub Copilot and a group without. The Copilot group finished 55.8 percent faster. Both studies are competently run randomized trials. Neither is wrong. They measured different kinds of work.

A bounded, well-specified, greenfield task is close to the ideal case for a large language model: no institutional memory required, a clear definition of done, nothing about the surrounding system left to misunderstand. Open-ended maintenance in a codebase a developer has spent years learning is close to the opposite case. The value the human brings is disproportionately in the parts that are hard to state as a prompt — which invariant not to break, which past attempt already failed, which stakeholder will complain about which edge case — and every minute the AI spends re-deriving context the human already had for free is a minute added to the ledger, not subtracted from it. Most vendor demonstrations, and a fair number of published benchmarks, look like the first case. Most of what a senior engineer actually does in a mature system looks like the second. METR is one of the few studies willing to point the instrument at the second kind of work, which is exactly why the result surprised people who had only ever seen the first kind measured.

The instrument that reads wrong

The gap between how fast work feels and how fast it actually was did not start with AI. A 2020 study of eighty-one developers at Microsoft, comparing automatically logged activity against those same developers’ self-reported sense of their own output, found only a weak relationship between the two — self-report and instrumented measurement were, in effect, answering different questions even before either side of the desk had a coding assistant. Developers have never been particularly well-calibrated instruments for reading their own throughput. AI did not invent that miscalibration; it raised the stakes attached to it, because for the first time an organization’s entire tooling budget, and often its headcount plan, hangs on trusting that same unreliable reading.

Part of the explanation is mechanical rather than purely psychological. When AI is allowed, a meaningful share of a session is spent watching code appear on screen, which is a visibly active state — something is clearly happening — even when the minutes that follow are spent reading, correcting, and re-prompting because the first attempt was not quite right. The visible portion of the work produces the felt sense of momentum; the invisible portion, verification, is exactly the part that does not register as effort while it is happening. A session can be net slower and still feel continuously productive from the inside, because the two halves of it are not weighted the same way by the person living through them.

This pattern has a name in economics, coined well before any of today’s coding tools existed. In 1987, the economist Robert Solow observed that computers had become visible everywhere in the American economy except in the productivity statistics — a technology adopted enthusiastically for over a decade before the aggregate output numbers showed anything for it. The lag turned out to be real: the gains eventually arrived, but only once organizations redesigned how the work itself was done, rather than simply typing the same processes faster. AI-assisted coding may be tracing the same curve at a much faster clock speed — a tool that is visible in every developer’s day, running well ahead of any aggregate evidence that the work it touches is actually finishing sooner.

What changed by 2026

METR reran a version of the same study in early 2026, and the headline number looked like vindication: the same population now estimated roughly an eighteen percent speedup with the newer generation of tools. Read uncritically, that is the story engineering leaders wanted the first time — the tools got better, and the humans finally noticed. METR’s own writeup complicates that reading. The researchers flagged a selection effect working against a clean interpretation: developers are increasingly unwilling to participate in a study, or in some cases to work at all, without AI access, which means the population willing to be measured is shifting toward people who already believe the tools work, independent of what the tools actually do to their completion time.

Both readings are credible, and they disagree — which is the honest, contested state of the evidence in 2026, not a data problem waiting to be resolved by picking a side. The underlying capability may genuinely be improving fast enough to erase a nineteen-point deficit in well under a year. The sample measuring it may also be self-selecting toward believers fast enough to produce the same-looking number regardless of what the tools actually do. A rigorous instrument built specifically to correct for perception bias now has to correct for a second bias that AI’s own ubiquity introduced into its subject pool — a genuinely new methodological problem, and one worth naming rather than smoothing over in a headline.

A checklist for valid measurement

Rather than treat one study as truth and the other as noise, it is more useful to extract what separated a result worth trusting from a survey number worth ignoring — because that same checklist is what separates real instrumentation from vibes at the level of a single team.

None of these rows require a research budget. They require deciding, before the next quarterly review, what a team would actually need to see to believe its own sentiment number — and then checking whether that evidence exists yet.

Checking sentiment against a session log

The comparison most teams are missing is not exotic. It is simply pairing the survey against the log: for the developers who say they feel faster, is there a matching record of session duration and eventual outcome — merged, reverted, abandoned — that the feeling can be checked against? Almost every team can run the survey. Very few can run the second half of the comparison, which is the only half that would actually answer the question.

None of this makes sentiment worthless. A team that feels miserable using a tool will find ways around it regardless of what a benchmark says, and morale is a real cost with real downstream effects on retention and quality. But sentiment is not a substitute for measurement, and treating it as one is how an organization ends up making large tooling and staffing decisions on the same kind of instrument that told sixteen experienced developers they were saving time while a stopwatch recorded them losing it. Knowing whether AI-assisted work is actually faster, better, or merely louder requires recording what happened — per session, per change, against an outcome nobody can rationalize after the fact — and checking the survey against that record the way METR checked forecast against actual. A team that skips this step has not measured its productivity. It has measured how good the tool is at feeling productive, which a well-built interface can produce even when nothing downstream got faster at all.