Measuring AI

An engineering manager opens the Copilot Business dashboard the morning before a quarterly review and finds a number worth screenshotting: acceptance rate up several points quarter over quarter, comfortably ahead of the benchmark GitHub publishes for comparison. It reads like progress, because acceptance is the metric the tool was built to report, and it updates in something close to real time. The slide goes in the deck. Nobody in the room asks the question the dashboard was never built to answer: of everything the team accepted this quarter, how much of it is still in the codebase, unmodified, three weeks later?

That second question is not a hypothetical gap — it is the exact seam between two AI-specific metrics that look almost identical in a slide deck and behave completely differently in a codebase. Acceptance fires the moment a suggestion is taken. Retention can only be answered by waiting, then going back to look. Most teams instrument the first, because a vendor ships it for free. Almost none instrument the second, because it means joining a decision made in an editor to a git history that has since moved on without it.

The layer underneath the acceptance number

Underneath acceptance sits a plainer, older layer of AI-specific telemetry: tokens consumed, dollars billed, broken down by session, by task, by engineer wherever a vendor console goes that granular. This is the layer this Library’s economics volume names the visible tip of the AI Cost Iceberg — a small, precisely metered figure sitting on top of a much larger, unbilled mass of review time, rework, and context rebuilding. The token bill is real and worth capturing at session grain rather than as a monthly aggregate; it is also, on its own, just a number. Cost without an outcome attached to it does not tell an engineering leader whether the spend was well placed. It tells them what the spend was, which is a smaller and much easier question.

That distinction — activity captured versus outcome connected — is the one this chapter keeps returning to, because it recurs at every layer above the token bill too. A session’s cost only becomes informative once it is divided by what the session actually returned, the same move this Library’s Session ROI concept makes for one sitting at a time. Acceptance rate turns out to be the same trap wearing a friendlier face: cheap to instrument, real-time, vendor-shipped — and, on its own, an activity metric answering a narrower question than it is usually asked to answer.

What acceptance rate actually measures

Acceptance rate deserves to be taken at face value for exactly what it is, not dismissed and not oversold. GitHub’s own documentation on Copilot usage metrics defines it plainly: the share of suggestions a developer takes, offered as a signal of whether suggestions are relevant and trusted. It is a real, useful, point-in-time measurement — a proxy for whether an AI suggestion was useful, judged at the only moment the tool can actually see, which is the instant it was offered. GitHub’s own framing does not claim anything more than that. The documentation is silent — not evasive, just silent — on what happens to that code afterward, because the acceptance event and everything that happens next live in two different systems that nothing joins by default.

The harder number: retention

Retention asks the harder, later, more honest question acceptance cannot: of what got accepted, how much is still there — unmodified or lightly modified — some fixed number of weeks on. A suggestion accepted at two in the afternoon has cleared exactly one bar: a developer, in the moment, judged it good enough to keep typing past. That bar says nothing about whether the same developer rewrites half of it in review, whether it survives the next refactor, or whether it is still recognizably the AI’s code a month later rather than a human correction wearing the AI’s commit message. This volume’s chapter on measuring quality develops the fuller churn and rework evidence behind that gap; retention is the same question asked one grain finer, at the level of a single accepted suggestion rather than a whole file or session.

Industry commentary aimed at exactly this gap makes the point bluntly: code accepted by an AI tool routinely gets deleted, refactored, or rewritten before it ever reaches production, which makes a raw acceptance count an unreliable proxy for anything durable happening downstream. That is not a reason to distrust acceptance telemetry. It is a reason to stop treating it as a finish line.

Two findings that do not agree

The evidence on what happens after acceptance does not point in one direction, and a chapter about measuring AI has an obligation to show the disagreement rather than reach for the tidier side. This volume’s chapter on measuring quality already documents the churn side of that disagreement in full — GitClear’s contested finding that code revised within two weeks of commit has been climbing since AI-assisted coding became common, and the direct counter-study that found no such general increase once AI involvement was verified by developers’ own admission rather than inferred from timing.

A separate 2026 study complicates the picture from a third angle entirely. A survival analysis of more than two hundred thousand code units across two hundred and one open-source projects found AI-agent-authored code carrying a 15.8-percentage-point lower modification rate and a sixteen percent lower hazard of being changed at all, compared with human-authored code in the same repositories over the same history. The authors are explicit about what this does to the popular assumption that AI-generated code is disposable — quickly merged, quickly ripped out: it does not hold up in their sample. Where the AI code did get modified, it skewed toward small corrective fixes rather than the larger adaptive rewrites human code more often received.

Both findings can be true at once, because they are not measuring the same population or the same clock. Open-source maintainers reviewing an agent’s pull request before merge are a different filter than a developer in an editor accepting an inline suggestion mid-keystroke, and a project’s full commit history is a different window than the two weeks a churn metric typically uses. The honest conclusion is not that one study is wrong. It is that retention depends heavily on the review discipline sitting between acceptance and the codebase — and that discipline is precisely the variable an acceptance-rate dashboard has no way to see.

One study finds AI code disappearing faster than it should. Another finds it outlasting the human code next to it. Both are real, both are recent, and both are looking at a different filter between acceptance and the repository.

On the retention evidence this chapter reviews

A common unit, and a common schema

A third grain sits above both of these: the session, one goal, one sitting, the unit this Library treats as the smallest slice of AI-assisted work with a real outcome attached. This volume’s next chapter mines session-level data directly — including a large study of real coding-agent sessions where most breakdowns between developer and agent showed up as effort and trust costs that a suggestion-level metric has no way to surface. The point worth holding onto here, before that chapter develops it fully, is narrower: acceptance, retention, and session outcome are three different grains of the same underlying question, and a measurement practice that only instruments one of them has chosen, by omission, which two-thirds of the answer it will never see.

What ties these grains together technically, at least at the lowest level, is a real and current standardization effort: OpenTelemetry’s GenAI semantic conventions. Still at Development maturity as of this writing, not yet a finished stable specification, the conventions define a common attribute set for instrumenting a call to a language model — which provider and model handled the request, how many tokens went in and out, cache hits, the operation type, chat completion or embedding or tool execution, and the span structure tying a sequence of those calls into an agent’s larger unit of work. It exists specifically to fix the default failure mode: every vendor building its own token dashboard on its own private schema, unable to be joined with anything else.

What the convention standardizes is genuinely useful and genuinely narrow. Fiddler AI’s own critique of the standard puts the limit bluntly: “It captures what happened. It does not assess whether what happened was good.” The GenAI conventions carry no attribute for a suggestion being accepted and none for a suggestion still being present thirty days later — that vocabulary belongs to the coding-tool layer sitting above the model call, and nothing has standardized it yet. The trace format is arriving before the outcome vocabulary is, which is the right order for a standard to mature in. It also means acceptance and retention will keep being measured in whatever bespoke way each tool chooses for a while yet.

What to instrument this week

None of this requires waiting for OpenTelemetry to reach a stable release, or for a vendor to ship a retention metric nobody has asked for loudly enough yet. Most of it is buildable this week against data a team already has.

1. Capture token and cost at session grain, not as a monthly total — the AI Cost Iceberg’s visible tip is only useful sliced fine enough to attach to an outcome later.
2. Pull whatever acceptance-rate telemetry the tool already ships, and label it accurately: an adoption signal, not a quality signal.
3. Pick a settlement window — thirty days is a defensible default — and run one blame-based join from accepted suggestions back to current file state to get a real retention number, even a rough one.
4. Where a bespoke pipeline is unavoidable, use OpenTelemetry’s gen_ai.* attribute names instead of inventing new ones, so the next tool that joins the same trace does not have to reconcile a private schema.
5. Report acceptance and retention on the same chart, always — neither number is trustworthy read alone.

The reason to be careful with all of this — and the reason it deserves its own honest caveat rather than a triumphant close — is that AI-specific telemetry is unusually easy to turn into surveillance. A token count is trivial to graph, trivial to rank by engineer, and trivial for a manager under adoption pressure to start pushing a team to maximize regardless of what it actually correlates with. DORA’s own account of AI as an amplifier of whatever an organization already is applies here without modification: a measurement culture that already trusts engineers will use acceptance and retention data to have a better conversation about tooling and workflow; one that does not will use the same two numbers to rank people. This volume turns exactly this scrutiny on the acceptance-rate proposal made earlier in this chapter — including its own — a few chapters ahead, in Healthy Metrics, Dangerous Metrics; every instrument this chapter recommends is worth reading against that later argument, not instead of it.