Session ROI

Monday morning, an engineering lead scans the weekend’s AI sessions the way a shift supervisor scans an overnight log: fast, looking for red. Thirty-one sessions ran while the on-call rotation slept. Twenty-eight merged clean, three were abandoned mid-goal for a few cents of tokens, and the total bill for the weekend was smaller than a single team lunch. By every number visible on the screen, the weekend was a success — cheap, mostly resolved, nothing on fire. The lead closes the tab and moves on to standup.

Ten days later, one of those twenty-eight merges gets reverted. It broke a downstream integration test nobody ran until the next deploy train, and untangling the fix costs a senior engineer most of an afternoon, because two other changes have landed on top of it in the meantime. Nobody connects the revert back to the Monday scorecard, because the Monday scorecard no longer exists as a live object — it was a snapshot, taken at the moment the session looked finished, and nothing in the tooling updates it when the session’s actual outcome arrives ten days late. The session was scored a win. On a ten-day view, it looks closer to a loss with a two-week payment plan.

What the dashboard already shows

This is not, at root, a missing-instrumentation problem. Most teams running AI-assisted sessions already have more visibility into the moment of completion than they know what to do with — merge status, session cost, duration, tool calls, error counts, all captured, all real-time, all genuinely useful for the question they answer. That question is narrower than it looks: did this session appear to finish successfully. DORA’s 2025 research frames the gap precisely. AI functions primarily as an amplifier of whatever organizational system it is dropped into, magnifying existing strengths and existing dysfunctions alike, and the return on AI investment comes from that surrounding system rather than from the tool itself. A session-scoring dashboard that only looks at the moment of generation is amplifying noise as readily as signal, because it has no way to tell the two apart until later.

Done is not the same as worked

Two recent findings sharpen the point. DX’s analysis of change failure rate in AI-assisted teams observes that a healthy-looking failure rate can conceal a high rework rate: when a team is quietly fixing AI-generated errors before they ever reach production, the failure metric stays clean while the effort the fix consumed disappears from the account entirely. A session that looks resolved because someone caught its problems off the books is not free — it is expensive in a place nobody is looking. Separately, an early-2026 academic study of more than thirty thousand agent-authored pull requests found a genuinely two-regime pattern: a large share merge almost instantly with no friction, but once a pull request enters an actual review loop, a meaningful fraction of agent-authored changes are simply abandoned mid-negotiation — “ghosted,” in the researchers’ term — after consuming real reviewer attention with nothing shipped to show for it.

Session ROI

Chapter 1 named the session the right unit of account for AI-assisted work — the smallest slice with a real outcome, sitting between prompts (too granular to mean anything on their own) and workflows (too aggregated to diagnose anything specific). This chapter names the metric that belongs at that layer: Session ROI, the value an AI session creates, measured against the time and money invested in it. The definition sounds obvious enough to be unremarkable. What makes it hard in practice is that neither half of the ratio is available at the moment the session ends. Cost settles fast — tokens and wall-clock time are known within minutes. Value does not settle nearly as fast. It settles when the change survives contact with the rest of the codebase, the review process, and production, and that can take anywhere from an afternoon to a month.

The Session Value Dashboard

A workable Session ROI practice needs more than a merged-or-not flag. The framework below — a Session Value Dashboard — breaks the ratio into five dimensions a team can populate from data it almost certainly already has, scattered across its repository, its review tool, and its AI usage logs. None of the five requires a new data source. What most teams lack is not the data; it is the habit of looking the five up together instead of stopping at the first one that answers quickly.

Durability deserves the most unpacking, because it is the dimension the industry already has a mature analogue for, just not at this grain. DORA’s own methodology treats change failure rate — the share of deployments that require a rollback or a hotfix — as a core delivery metric, and its guidance now names a companion metric, deployment rework rate, specifically to catch the failures that get quietly patched before anyone calls them a rollback. Session ROI borrows the same instinct and applies it one level down: an outcome tag written at merge time is a change-failure-rate proxy taken too early. Revisiting it at a fixed window — thirty days is a defensible default, long enough for most integration and review problems to surface, short enough to stay actionable — turns a single snapshot into the delayed-settlement measurement the ratio actually needs.

An outcome tag written at merge time is a change-failure-rate proxy taken too early.

On why durability, not merge status, settles the ratio

Downstream cost is the other row worth sitting with, because it is where the invoice and the real cost diverge furthest. In one ten-thousand-developer dataset, teams using AI completed more tasks and nearly doubled pull-request throughput while review time grew 91% — generation got faster and verification absorbed the gain, largely invisibly, because review time is billed to the reviewer’s calendar, not the session that created the work. Quality debt compounds the same way: large-scale analyses of AI-era repositories report code duplication rising and the refactoring share of changes falling since adoption began, though the methodology behind those specific figures is contested even as the direction is not. Both are downstream cost by another name — work a session generates that lands on someone else’s ledger, weeks after the session itself has been filed away as done.

Two ways to miscount

Two mistakes recur once a team tries to operationalize this. The first is measuring too early — treating the merge-time outcome tag as final, which is exactly the Monday-morning illusion this chapter opened with. A session is not done paying for itself until its output has passed through the parts of the system that catch what generation alone cannot: integration tests that run on a delay, a reviewer who finally reaches the queue, a production load the sandbox never simulated. Closing the book at merge time is not measuring ROI; it is measuring enthusiasm, and Chapter 1 already showed how unreliable enthusiasm is as an instrument — the same METR trial that found experienced developers measurably slower while feeling faster is a warning against trusting how a session felt in the room over what it did in the following weeks.

The second mistake is measuring too coarsely — averaging Session ROI across every session in a month and reporting the mean as though it described a typical session. It is a plausible pattern, illustrative here rather than a specific verified statistic, that AI session cost and failure are not distributed evenly: a small number of long, unfocused sessions — the kind that wander through a codebase without a clear stopping point, sometimes called rabbit holes — tend to account for a disproportionate share of both the spend and the eventual reverts, while the typical session is cheap and fine. A mean blends the rabbit holes into the median and hides both. The correction is the one any right-skewed distribution calls for: rank instead of averaging. Look at the top decile by cost before looking at the mean, and check whether it overlaps with the bottom decile by durability.

Building the dashboard, this week

1. Tag every session’s outcome at least three ways: merged, merged-then-reverted, abandoned or blocked. Version control already answers the first two; the third just needs a place to be written down.
2. Pick a durability window and hold to it. Thirty days is a reasonable default, long enough for most integration and review problems to surface. Revisit outcomes at the end of it, not just at merge.
3. Track downstream cost as its own line rather than folding it into the original session’s cost — review time, rework commits, and any follow-up session that exists only to fix the first one.
4. Rank sessions by cost before computing an average. The top decile usually explains more about AI spend than the mean does.
5. Compute Session ROI per cohort — by task type, by engineer, by tool — before reporting one company-wide number. A healthy median can sit right next to an unhealthy tail, and a single average erases the difference that matters.

None of this requires a new tool, and it does not require abandoning the merge-time view — that view is still the fastest signal available and should not be discarded. It requires treating that view as provisional, the way a surgeon treats a patient’s condition at the end of an operation as provisional until recovery confirms it. The chapters ahead build on the same instinct: the debt that accrues when a session has to re-earn context it already paid for once, and the fuller accounting of everything sitting below the token bill that this chapter’s downstream-cost row has only begun to sketch.