Engineering Signals

Three weeks after a retry-logic change ships, it turns out to be the cause of a production incident. The engineer who ran that AI session has moved on to a different part of the codebase since. The terminal window is long closed. Slack has one line: “agent handled the retry logic, tests green, merging.” What actually happened during the forty minutes the session ran — which approach the agent tried first and abandoned, which files it read before it touched anything, whether the test suite that supposedly went green ran against the final version of the code or an earlier one — is gone. Not archived, not summarized, not slow to retrieve. Gone, the way a phone call is gone the moment you hang up. The postmortem ends up reconstructed from a diff and a memory, which is the same quality of evidence engineering teams abandoned for their own production systems a decade ago.

This is not a story about a careless team. It is the default outcome of how most AI-assisted sessions run today: a terminal, a chat pane, or an IDE extension streams a rich sequence of prompts, tool calls, file edits, test results, and errors — and when the session ends, whichever part of that stream was not explicitly saved simply stops existing. The scrollback clears. The conversation history ages out of context. Nobody made a decision to delete the record. Nobody made a decision to keep it either.

What the session already generated

The instinct is to treat this as a data problem — as if the fix requires bolting new instrumentation onto every AI tool a team uses. It does not, and that is the useful part of the argument. A session already generates almost everything worth keeping, as a byproduct of doing the work. Every tool call an agent makes is a discrete, timestamped, structured event: read this file, run this command, produce this diff, hit this error. None of it needs to be invented. It needs to be captured before it disappears.

The evidence that this exhaust data is diagnostically valuable already exists, precisely because someone bothered to capture it. Anthropic’s account of its own multi-agent research system found that token and tool-use telemetry — the same category of signal a session throws off constantly — explained roughly 80% of the variance in agentic task performance, more than model choice or prompt design. That finding was never available to a team that discards its tool-call logs at the end of every session; it exists only because someone in that pipeline decided the telemetry was worth keeping.

The capture problem, not the data problem

This reframes the failure. AI-assisted development is not under-instrumented in some abstract sense — nothing in most teams’ toolchains has been assigned the job of persisting what already streams past. A hook that fires on tool use, a thin wrapper around the CLI, a proxy sitting in front of the model API, a harness-level capture layer: any of these can do it, and none require the agent to behave differently or the developer to do extra work. Claude Code itself ships a documented lifecycle of hookable events — session start, before and after each tool call, notification, session end — that map almost one to one onto the events worth keeping. The capability to capture is not exotic or unbuilt. It is sitting, mostly unused, in tools teams already run every day.

What capture is not

It is worth being precise about what “cheap to capture” does not mean, because the claim collapses if it is overstated. The events themselves cost nothing extra to generate — the agent was always going to call that tool, read that file, run that test. What is not free is the small amount of engineering required to build the capture layer once: deciding where events get persisted, scrubbing secrets and credentials out of tool arguments before they land in a durable store, deciding a retention policy so the record does not become its own liability, and keeping the hook itself thin enough that it never becomes the reason a session feels slow. None of that is a research problem. It is closer to the work of adding structured logging to a service that previously logged to stdout and hoped — bounded, well-understood, and done once per organization rather than once per session.

A session as an event stream

The conceptual move worth making is to stop treating an AI session as an ephemeral conversation and start treating it as a production event stream — the same category of thing a web request becomes once a distributed system emits a trace for it. A request touching a dozen internal services used to be as invisible, end to end, as an AI session is today; nobody could see the whole path, only fragments scattered across different logs. The industry did not solve that by asking engineers to remember requests better. It solved it by instrumenting the request path once, structurally, and letting every later question about what actually happened be answered by querying the record instead of reconstructing it from memory. A session deserves the same treatment, at session grain instead of request grain.

Monitoring answers the questions you already thought to ask. Observability answers the ones you did not.

On the monitoring/observability distinction

That distinction — reacting to failures you already expect, versus being able to ask a system a question you did not anticipate — is what the software-observability literature draws between monitoring and observability. An AI session that only surfaces a pass/fail status and a final diff is monitored, at best; it is not observable. Nobody can go back and ask which of last month’s forty sessions touched the payment module, or how many times the agent read the same file before committing to an approach, because that question was never anticipated at capture time. With structured events instead of a transcript, it does not need to have been.

The Telemetry Schema

What belongs in that structured event stream is not a matter of taste. It follows directly from the session lifecycle every AI-assisted change already passes through, and each event type answers a specific question later — about cost, about workflow, about quality — that a transcript alone cannot.

Nothing in that table requires a new measurement to be invented. Every row is something the session already does; the schema just insists that each of those events gets written down instead of scrolling past. Once it is written down, the questions the rest of this volume depends on — cost per merged change, where review time actually goes, which sessions belong in a quality ledger — stop being retrospective interviews conducted weeks later against a fading memory, and start being queries against a table that was there the whole time.

What a captured session looks like

The observability precedent

None of this is a novel idea, and it should not be presented as one. The industry made this exact move with distributed systems a decade earlier. Google’s Dapper paper, published in 2010, described the tracing infrastructure built after a single search request touching dozens of internal services became too complex to debug from logs and intuition alone; the fix was to instrument the request path structurally, once, and query it afterward. Structured logging and distributed tracing went from a research proposal to an industry default over roughly a decade, and the standards effort now underway for AI agents — the OpenTelemetry project’s GenAI semantic conventions, defining what a model or tool-call span should record — is the same convergence happening again, this time for agent sessions instead of microservice requests.

The DORA research group’s AI Capabilities Model makes the same point from the organizational-maturity side. Teams with what it calls a healthy data ecosystem — internal data that is high-quality, accessible, and not fragmented across systems — see AI’s positive effect on outcomes amplify; teams without one see the same tools produce noise instead. Session telemetry, captured consistently, is exactly the kind of internal data that model is describing: not a dashboard bolted on afterward, but a record the organization’s own AI-assisted work generates and keeps. It is expensive to build once and, unlike most engineering investments, it gets more valuable the longer it accumulates rather than less.

The expensive part of this move, historically, has been building the capture layer the first time — deciding what to hook, where to persist it, how to keep it from adding latency or leaking secrets. That work is done exactly once. Chapter 10 picks up from here: not whether to capture, which this chapter has argued is close to a solved problem, but what an observability layer built on top of that captured stream actually needs to expose to be useful to an engineer who was not in the room when the session ran.