The Case for AI Observability

Ask most engineering leaders what their team spent on AI coding tools last month and they can usually produce a number within a day — pulled from a vendor dashboard, rounded to the nearest thousand dollars. Ask the same leader which sessions produced that spend, which team ran them, and which model handled the bulk of the tokens, and the day stretches into a week. The invoice is precise. Everything upstream of the invoice gets reconstructed from memory, chat scrollback, and whoever remembers which project was running heavy AI usage in the second week of the month.

This is not a story about a team that failed to build good habits. It is the default state of AI-assisted engineering at most organizations in 2026: the billing relationship with the model vendor is instrumented to the token, and the engineering relationship — who ran what, on what, with what result — is not instrumented at all. The gap does not close as usage grows. It widens, because more sessions simply means more untracked activity to reconstruct later.

The month-end reconciliation problem

Picture the conversation that happens in most finance reviews once AI coding spend crosses a threshold someone notices. The number is up — twenty percent, forty percent, sometimes more — and finance wants to know why. Engineering’s honest answer is usually a shrug dressed up as confidence: more people are using it, on more things, probably a few bigger sessions. That answer is not wrong. It is also not one finance can act on, because it carries no breakdown by team, by project, by model tier, or by whether the spend produced anything that shipped. Without real-time token tracking joined to who ran what, the only view available is the one the vendor’s invoice provides: a lagging, monthly, organization-wide total that arrives weeks after the decisions that drove it.

The previous chapter in this volume, Engineering Signals, argued that a session is not a black box — it is a rich, capturable stream of events: which files were touched, which tools were called, how the work ended. That argument was about what could be captured. This one is about why an organization should actually build the layer that captures it, and what breaks when it does not.

Three ways the picture breaks down

Three specific gaps recur across teams that have not built this layer, and each compounds the other two.

• No real-time token tracking. Cost visibility exists only at the grain the vendor bills at — usually organization-wide, monthly. A team cannot answer what it spent this week, and on what, without manually reconciling invoices after the fact; by the time the number arrives, the sessions that produced it are long over and the context for judging whether they were worth it is gone.
• No maintained pricing registry. Model pricing and lineups change on a cadence closer to quarterly than annual, and a raw token count is meaningless without a current dollar-per-token figure attached to the specific model that generated it. A cost dashboard that hardcodes last quarter’s rates, or cannot tell a cheap model from an expensive one, breaks the moment a team switches models or a vendor reprices a tier — and it breaks silently, producing a confident, wrong number.
• No unified telemetry. Session events, tool calls, and outcomes each live in their own system, unjoined. Even a team with excellent token tracking and a current pricing table still cannot say whether a given dollar of spend produced a merged change, a revert, or nothing — because the system that knows the cost and the system that knows the outcome have never been introduced to each other.

Each gap is survivable alone. Together, they explain why most organizations can recite last month’s AI bill to the dollar and still cannot say whether the bill was money well spent. And each gap gets worse, not better, as a team matures its usage — a team running one model through one tool has an easy time eyeballing its way past all three problems; a team running several models across several tools, with engineers free to pick whichever fits the task, cannot eyeball its way past any of them.

Observability as precondition, not dashboard

The discipline that closes these gaps already has a name, borrowed from a decade of site-reliability practice: observability. Extending it to AI-assisted engineering means treating a session the way SRE practice treats a request — as an event with a full trace, not a line item on an invoice. The industry has begun formalizing what that trace should contain: OpenTelemetry’s GenAI semantic conventions, still maturing as of 2025, define a standard shape for capturing model calls, token counts, and outcomes, so instrumentation from different tools and vendors can be joined instead of reinvented per integration. That a standards body has stepped in to define this at all is itself a signal — ad hoc, per-vendor capture was the default failure mode the standard exists to fix.

The payoff for building this layer is not hypothetical. Anthropic’s own published analysis of its multi-agent research system — the finding, cited earlier in this volume, that token usage alone explains roughly 80% of the variance in agentic task performance — is possible only because the organization had granular token and cost telemetry to analyze in the first place. A specific, falsifiable claim about how AI-assisted work actually behaves is downstream of instrumentation. Without it, an organization is left with impressions: it felt more effective this quarter, tokens seemed higher, the team seemed faster. This volume’s second chapter already showed how unreliable those impressions can be, even from experienced engineers rating their own work.

The Unified Observability Stack

A useful way to organize the build is as a stack of four layers, each depending on the one below it and each answering a different question.

The stack is ordered by dependency, not by difficulty. Capture is a prerequisite for cost attribution; capture and cost attribution together are prerequisites for correlation; and none of the three matters to a working engineer unless something at the presentation layer surfaces it at the moment a decision is being made. Skipping a layer does not simplify the system. It just means the layer above it is quietly wrong.

Where the tooling stops

Most of the tooling ecosystem built around AI-assisted engineering in 2026 — engineering-intelligence platforms, AI-specific cost-analytics products, the growing set of OpenTelemetry-compatible LLM observability tools — concentrates on the bottom two layers of this stack. That concentration makes sense: capture and cost attribution have an emerging standard to build against, a clear unit of measure, and a buyer with an obvious pain point, the invoice. They are also the layers a vendor can sell without needing access to a customer’s git history, review system, and incident log all at once.

It captures what happened. It does not assess whether what happened was good.

Fiddler AI, on the limits of OpenTelemetry-based AI observability

That gap is the correlation layer, and it is where the tooling landscape is thinnest — not neglected, but comparatively immature, and not owned by any single vendor as of this writing. The clearest illustration is a finding this volume returns to more than once: one widely cited dataset found AI adoption correlated with pull-request throughput nearly doubling while review time grew 91% and delivery metrics stayed flat. That correlation was discoverable only because someone joined AI-usage metadata to review-system timestamps by hand. Most teams do not have that join running continuously, let alone a vendor supplying it as a standard report — which means the review bottleneck this volume names later stays invisible for most organizations until it is already a crisis.

The gap shows up in smaller, more everyday ways too. A team that switches its default model mid-quarter — a common move, given how often pricing and capability shift — gets a clean before-and-after view of dollars spent at the capture and cost-attribution layers, and no equivalent view of whether the new model changed the merge rate, the revert rate, or the time a change spent in review. The tooling answers the question a vendor asks first (what did we spend) and stays silent on the question an engineering leader actually needs answered (did the switch help), because that second question requires data the cost-analytics tool was never given access to.

The minimum viable version

None of this requires a platform migration or a new procurement cycle to start. A minimal version of the stack is buildable on top of whatever session capture a team already has, in three concrete moves: tag every captured session event with the model that generated it and its raw token counts; maintain a small, explicitly versioned pricing table — a plain mapping of model identifier to dollars per million input and output tokens, updated whenever a vendor reprices, which happens on something close to a quarterly cadence across major providers; and store one join key per session — a commit hash, a pull-request number, a ticket ID — that lets a later query connect cost to outcome without a special integration project. The third move is the one teams skip, and it is the one that makes every other chapter in this volume computable instead of estimated.

Why this precedes the rest of the volume

Every economic argument the rest of this volume makes assumes this stack, or something like it, exists. Session ROI needs a cost figure and an outcome figure for the same session. Cost per outcome needs the same join, aggregated across sessions. The review bottleneck needs cost data correlated with review-system timestamps. The quality ledger needs a change’s AI provenance linked to what happened to it months later. None of these are new metrics bolted onto a mature practice. They are queries against a dataset that, for most organizations today, does not yet exist in queryable form.

That is the case for AI observability, stated plainly. It is not a dashboard layered on top of AI-assisted economics after the fact. It is the precondition for that economics being a discipline at all, rather than a set of opinions restated at every retro.