Measuring Learning

Six months after an engineering org gates incident closure on a runbook or decision-record update — the structural fix this Library’s ninth volume argued for, so that a finding cannot simply be forgotten by the person who found it — a VP asks a plain question in a quarterly review: are we actually learning faster, or just closing tickets faster? The velocity chart is exactly where it always is, up and to the right. Deploy frequency is up. AI-assisted throughput is up. Nobody in the room can point to a chart that answers the question that was actually asked, because nobody has ever built one.

That is not a failure specific to this team. It is the same gap this volume’s second chapter, on the DORA AI Capabilities Model, already found and left standing: individual and team output can rise sharply without the organization itself getting measurably better at anything. This chapter takes that gap and asks the narrower, harder question it implies — if an organization is genuinely learning, not merely producing faster, what would that look like in data a team could actually collect?

What the dashboards already show

The 2025 DORA report’s central claim is that AI functions as an amplifier — it magnifies an organization’s existing strengths and dysfunctions rather than replacing either. One of the concrete shapes that amplification takes, reported by Faros AI’s analysis of a ten-thousand-developer dataset, is individual output rising while organizational metrics stay flat: roughly 21% more tasks completed and pull-request throughput nearly doubling, while delivery metrics held steady and review time grew 91%. Every one of those figures describes activity — more work generated, more work reviewed, more work merged. None of them describes whether the organization got better at avoiding the mistakes it has already made once.

The structural loop’s missing instrument

Volume IX’s chapter on learning organizations drew a sharp line between an informal loop — a finding that changes practice only if a specific person remembers to change it, before a deadline or a team change intervenes — and a structural loop, where the artifact update is part of the incident’s own definition of done and runs whether or not anyone remembers to ask. That chapter’s argument was mechanical: build the wire, don’t rely on memory to carry the current. This chapter’s argument is that building the wire and knowing it is carrying current are two different accomplishments, and most organizations that have done the first have not done the second.

The honest reason a team would struggle to answer the VP’s question is not laziness. Measuring organizational learning is a genuinely less developed practice than measuring delivery velocity, and the field’s own best current answer says so. DORA’s guidance on measuring a climate for learning recommends, as its primary instrument, a validated employee survey — asking teams to rate agreement with statements like “learning is the key to improvement” and “once we quit learning we endanger our future.” That approach is credible and well-tested as a predictor of delivery performance. It is also, by DORA’s own framing, a measure of perceived climate, not a quantitative signal that could distinguish a team that is genuinely learning from one that is simply moving through more tickets per week and reporting high morale while doing it.

Organizations vary considerably in the rates at which they learn.

Linda Argote, on organizational learning curves

Three candidate signals

None of what follows is a direct measurement of organizational learning — nothing in the current literature is. Each is a leading indicator, borrowed from data most engineering organizations already generate for other reasons, that would move in a specific direction if the structural loop from Volume IX were actually running, and would sit flat or drift the wrong way if it were only running on paper. Treat all three as proxies to be checked against each other and revisited, not as a scorecard.

The first is the recurrence rate of the same root cause across incidents. A learning organization should see a given tagged root cause — a retry loop with no backoff, a specific class of race condition, a specific stale-config failure — show up less often per N incidents over time, not because that failure mode became impossible, but because the fix and the reasoning behind it actually propagated. Most organizations running any postmortem process at all already have the raw incident data to compute this. What most of them do not have is a root-cause taxonomy applied consistently enough to trust the count — a preliminary study of thirteen incident reports and seven practitioner interviews across multiple organizations with established postmortem tooling found that even there, translating a named lesson into a measured reliability improvement was, in the researchers’ own words, “not straightforward.” The recurrence rate is real and computable. Getting a taxonomy disciplined enough to make it mean something is the actual work, and it is worth naming as work rather than assuming it already exists.

The second is the closure rate and closure time of postmortem action items specifically tied to a durable-artifact update — a runbook, a decision record, a documentation page — as distinct from action items in general. This is the direct measurement companion to Volume IX’s proposed fix: gating incident closure on the artifact update. If that gate is actually in force, the subset of action items that touch a durable artifact should close at a rate approaching the rate for any other action item, and the median time from “finding discovered” to “artifact reflects it” should be trending down, not accumulating as a quietly growing backlog. Google’s own SRE workbook makes the case for tracking action items as a formal, monitored process rather than trusting memory precisely because the informal version reliably fails; the refinement this chapter adds is narrower — track the durable-artifact subset separately from the count of all action items closed, because a team can close every code-fix item on schedule while the runbook that should have caught the bug in the first place sits untouched, and a single combined closure rate would never show the difference.

The third is time- or cost-to-competency: whether a new engineer, or a new AI session, starting fresh on a familiar problem type reaches a working solution faster and more cheaply than the session that first ever faced that problem, given the knowledge infrastructure Volume IX’s capstone argued an organization needs to build. This is the most speculative of the three and the hardest to instrument cleanly, but it is also the one place AI-assisted engineering makes a genuinely new measurement possible that was previously very hard to isolate: a per-session cost and duration record, tagged by problem type, is something tools already capture where a comparable record of one engineer’s ramp-up rarely exists in comparable form. The academic grounding for the general idea is old and well-established — Argote and Epple’s work on organizational learning curves found large productivity gains from experience, but also that organizations vary considerably in how fast they capture them, degraded by factors like employee turnover and what the literature calls organizational forgetting. The honest catch is that having the infrastructure does not guarantee the transfer happens: Szulanski’s research on internal knowledge “stickiness” found that best practices often fail to move within a single firm even when nothing prevents it on paper, because the causal ambiguity of why something worked and the recipient’s own capacity to absorb it matter as much as whether a document exists. A falling cost-to-competency curve is a genuinely good sign. A flat one does not prove the knowledge layer failed — it might simply mean nobody has checked whether the sessions being compared were actually comparable.

What the data would need to show

What to build

1. Agree on a root-cause taxonomy before trying to trend recurrence — a recurrence rate computed against inconsistent tags is noise dressed as a metric.
2. Split action-item closure into two counts, not one: all items, and the subset tied to a durable artifact. Report both; only the second one answers this chapter’s question.
3. Timestamp the artifact update itself, not just the postmortem’s close date, so the finding-to-fix interval is measurable rather than assumed.
4. If sessions are tagged by problem type, compare cost and duration across occurrences of the same type before and after the relevant runbook or decision record existed.
5. Report all three signals next to the velocity metrics they are meant to check, not on a separate dashboard nobody opens in the same meeting.

None of this settles the question this volume’s second chapter raised — it gives that question a place to land. An organization whose AI-assisted throughput keeps climbing while its repeat-root-cause rate holds steady, its durable-artifact closure time keeps growing, and its cost-to-competency shows no improvement on familiar problem types is not a counterexample to the AI Capabilities Model’s amplifier finding. It is that finding, made specific and checkable: the amplifier is running, and there is nothing underneath it yet worth amplifying. Velocity climbing while these three signals stay flat is not a milder version of the problem. It is the clearest version available of an organization getting faster without getting better, which is exactly the distinction the dashboards were never built to make.