Session & Workflow Analytics

An engineering director pulls up the quarterly delivery report before a leadership review. Pull request cycle time is flat. Merge rate is up. Nothing in the report explains why three separate engineers have independently complained, in the last two weeks, that “the AI keeps going in circles” on anything touching the billing service. The report is not wrong. It is simply blind to the thing everyone in the room actually experienced — not the pull request, but the twenty minutes before it existed, when an agent tried an approach, watched it fail, tried an adjacent approach, and eventually landed somewhere serviceable after paying for all three attempts.

That twenty-minute stretch has no name in most engineering-metrics vocabularies, because nothing in the traditional toolset was built to see it. Commits, pull requests, sprints, and tickets are all units built around a human cadence of work — checkpoints a person chooses to create when something is ready to be looked at. AI-assisted engineering introduces a new unit that sits underneath all of them: the session, one continuous stretch of work between a person’s prompt or goal and the point the AI considers that goal done, however many tool calls, file edits, dead ends, and course corrections happened in between. A session usually produces a commit. It is not the same thing as one.

A unit the old grain cannot resolve

The clearest evidence that something real happens inside a session, invisible at commit grain, comes from the first large behavioral studies of how these interactions actually unfold. An analysis of 11,579 real-world AI-assisted IDE sessions found that developers rarely specify a task once and receive a finished answer — instead, a session is progressive specification: instructions get refined turn by turn, developers redistribute comprehension and validation work onto the agent mid-session, and plans get externalized into throwaway artifacts that never survive to the eventual commit message. Everything the study is actually measuring happens between the first prompt and the last accepted diff. A commit-level metric sees only the destination.

A separate large-scale study of CLI-based coding-agent sessions makes the concealment more concrete. Across six thousand sessions and over 355,000 tool calls, only 44% of the code an agent actually wrote survived into the developer’s final commit — and usage split into a strikingly bimodal pattern, with agents writing nearly all the committed code in 41% of sessions and developers writing essentially all of it themselves in 23%, with comparatively little in between. A commit-level view of that same population sees a healthy 44% acceptance figure and nothing about the shape underneath it: two very different kinds of sessions, averaged into one number that describes neither.

What a session-level view sees that a commit-level view cannot

This Library already has a name for judging one sitting’s worth against its cost — Session ROI, introduced in Volume I, asks what a given session returned relative to what it consumed. Session and workflow analytics is the discipline built on top of that same unit, but aimed at a different question. Where Session ROI asks whether one session was worth it, session-level analytics asks what, across many sessions, reliably predicts whether a session will be worth it — and where, inside a typical session, the cost actually concentrates. Those are workflow-level questions a per-session scorecard cannot answer on its own; they require treating the session as a row in a table with many other rows, not as an isolated verdict.

Volume V’s account of the AI session lifecycle — plan, prompt, generate, review, refine, commit, deploy, learn — gives that internal structure a name stage by stage. Session-level analytics is what happens when a team stops treating those eight stages as a description of one session and starts treating them as columns it can aggregate across a hundred sessions: how much of total cost sits in refine versus generate, which task categories reliably clear review on the first pass, which ones cycle through refine three or four times before anything ships. None of that is visible from a pull-request queue, because a pull request is what a session produces after the internal cycling is already over.

Three analytical questions the session grain newly enables

The first question — session length and cost distribution — is where the long tail lives, and Volume V’s chapter on workflow friction already describes the shape that tail usually takes without needing a session-analytics term for it: sessions that stall on edit thrashing, a search that never converges, or a run of cascading tool failures each one making the next more likely. A long, expensive session is not automatically a bad one — sometimes the task really was that hard, and the cost is real work rather than waste. What a session-level distribution adds is the ability to tell the two apart at scale: a task category whose sessions are reliably long and expensive is underspecified or genuinely difficult; a task category whose sessions are long, expensive, and specifically show the thrashing or spiraling shapes that chapter names is a friction problem worth fixing in the harness, not the task.

The second question, cost-to-outcome ratio, is the session-level extension of the token and cost telemetry this volume’s earlier chapter on measuring AI establishes as the ground truth layer underneath all of this — a session’s cost is meaningless as a standalone figure and only becomes informative once it is divided by what the session actually returned, which is the same move Session ROI already makes for one session at a time, applied here across a population of them.

The third question is the one this chapter’s hook gestures at most directly, and it is a genuinely open research question rather than a settled one. Volume V’s chapter on workflow patterns names five recurring session archetypes — plan-first, test-first, iterate, explore-first, greenfield — and is explicit that no rigorous, publicly published study directly compares cost, duration, and quality across all five head-to-head. The one adjacent finding that does exist, and complicates rather than confirms the intuitive story, comes from research on test-driven agentic development: instructing an agent to simply “follow TDD” as a plain procedural rule made outcomes worse, raising regressions from a 6.08% baseline to 9.94%, while giving the agent a structural aid — a map of which tests a change could actually affect — cut regressions to 1.82%. Naming a workflow pattern and actually equipping a session to execute it well turned out to be two different jobs. Whether plan-first sessions outperform dive-in sessions, and under which conditions, remains a question worth asking of a team’s own history rather than one the published literature currently answers for anyone.

How thin the evidence base actually is

It is worth being direct about this rather than implying more rigor than exists. Session-level and workflow-level analytics for AI coding agents is a genuinely young field as of 2026. The three studies cited above are the strongest currently available, and each comes with real limits: the 11,579-session behavioral study drew its data from one chat-logging tool used by developers who opted into committing their conversation history to public repositories, which is a specific and self-selected population, not a cross-organizational sample. The 6,000-session CLI study and the 20,574-session misalignment study are both single-team academic analyses, released as preprints in 2026, not yet independently replicated. None of the three is a randomized or controlled comparison of workflow shapes — all three are observational, describing what happened rather than establishing what caused what. Compare that to the decade-plus of validation behind commit- and PR-level metrics like the DORA four keys, and the honest description of session-level analytics is a promising, active area of inquiry, not a mature discipline with settled findings. Most of what practitioners currently have beyond these few academic studies is vendor telemetry and single-organization case evidence — useful for generating hypotheses, not for treating any specific number as a benchmark to hit.

What to build before what to buy

A team does not need a vendor platform to start asking these questions of its own history. The three ingredients are the ones already named above: a session boundary (start and end, however roughly drawn), a cost figure per session (this volume’s measuring-AI chapter covers how to make that figure trustworthy), and an outcome tag per session, even a coarse one. Everything past that — the distribution, the cost-to-outcome ratio, the workflow classification — is aggregation over data most teams already have and are not yet joining. The scarce resource is not instrumentation. It is the decision to treat many sessions as a dataset instead of many isolated stories.

None of this stays confined to one person’s sessions for long. The moment a team has enough sessions tagged to answer whether its default workflow pattern is actually its cheapest one, it also has enough data to ask the same question across people — whether one engineer’s sessions cluster around plan-first and another’s around iterate, whether that difference tracks with anything about outcomes, and whether a hard-won workflow lesson from one person’s history could have saved another person’s afternoon if anyone had been looking across both. That is a different kind of question than anything a single session, or even one person’s session history, can answer on its own. It is where this volume goes next.