Measuring Decisions

Two questions get raised in the same project, in the same week. The first — should the new ingestion service retry with backoff or fail fast — gets settled inside a single AI session, in about twenty minutes, by an engineer who reads the payments team’s old reasoning off the decision graph, agrees with it, and moves on. The second — should the team take a dependency on a vendor’s new embeddings API or build the equivalent in-house — takes six weeks. It survives two design reviews, three Slack threads that each restate the same three arguments, a stalled ticket, and an escalation email before anyone commits to an answer. Both decisions get written to the same decision graph this Library’s previous chapter described. The graph holds both records faithfully, with the same fields, the same lineage support, the same "why" a click away. It has nothing to say about which of the two processes that produced them the organization would actually want more of.

What the graph doesn’t say

Volume IX’s Decision Archives chapter solved a real, specific problem: whether a decision, once made, can be found again by someone who was not in the room when it happened. That is a discovery problem, and building a decision graph — decisions as queryable records, linked to what they touched and what they superseded — answers it. It says nothing about the process that produced any single decision inside the graph. A twenty-minute decision and a six-week decision look identical once they are both persisted rows with a "what" and a "why" attached. So does a good decision and a decision that gets quietly walked back three weeks later. Capture and retrieval were the problem the graph was built to solve. Whether the decisions flowing into it are fast, whether they are good, and whether the organization is revisiting them at a healthy rate are three different questions, and none of them were the ones Decision Archives was answering.

A discipline that already exists, one level up

The idea that decision-making itself is a process worth measuring, separately from whatever the decision was about, is not new and did not originate in software. Bain & Company’s decision-effectiveness research, drawn from a survey of nearly 800 companies, found decision effectiveness and financial results correlating at a 95 percent confidence level or better across country, industry, and company size — and named four distinct dimensions worth tracking separately: decision quality, speed, yield, and the effort spent getting there. That research measured organizations through surveys, not through timestamped software artifacts, and it has nothing to say about repositories or AI sessions. But it establishes something this chapter leans on directly: quality and speed are not the same axis, an organization can be strong on one and weak on the other, and measuring them separately is itself a defensible, previously validated idea — just one that has never, to this Library’s knowledge, been applied to a decision graph built from AI-assisted engineering sessions.

That gap is what this chapter tries to close, honestly and provisionally. None of the three metrics that follow has DORA’s pedigree — years of large-sample survey work behind an industry-standard definition everyone measures the same way. What they have instead is a real substrate to measure against for the first time: a decision graph with exact timestamps, supersession edges, and a linked trace for every decision, all of it already described in Volume IX rather than invented here. This chapter is a proposal built on that substrate, not a report on an established practice, and it should be read that way throughout.

Decision velocity

The most direct analogy is also the most familiar one in this volume: lead time for changes, one of the four keys this volume traces from DORA’s software delivery research, measures the time between a change being committed to version control and reaching production. Decision velocity, as proposed here, is the same shape of measurement applied to a decision instead of a commit — the time between a question entering the record and a decision being committed against it. What makes this newly computable, rather than merely conceivable, is that the substrate already carries both endpoints. A decision graph built the way Volume IX described links every decision to the trace and session that produced it, and a trace already has a start time. The gap between when a question first shows up in a session and when a decision gets persisted against it is sitting in data the graph already collects, without new instrumentation, for the first time simply because AI sessions leave a timestamped record that a Slack thread or a hallway conversation never did.

The honest risk sits right next to the appeal. A team that starts rewarding fast decisions, full stop, is optimizing exactly the kind of proxy this volume’s Chapter 10 examines directly under Goodhart’s law — a measure that, once it becomes a target, stops describing the thing it was meant to describe. Gary Klein’s premortem technique, in which a team deliberately imagines a project has already failed and works backward to surface objections before committing, exists precisely because some of the most valuable decision-quality work looks, from a stopwatch, like delay. A velocity metric applied without judgment would flag a deliberate premortem the same way it flags six weeks of nobody owning the follow-up — as slow, and therefore bad. Decision velocity is worth tracking. It is not, by itself, worth optimizing.

Decision quality

This is the harder metric, and it is worth admitting that plainly before proposing anything. Annie Duke’s distinction between decision quality and outcome quality — what she calls "resulting," the habit of judging a decision by what happened afterward rather than by the reasoning available at the time it was made — is the correct objection to any naive attempt to score decisions by their results. A good decision can still turn out badly on luck alone; a careless one can still turn out fine. Neither a decision graph nor any other software artifact can directly observe the quality of reasoning inside a session the way a trained facilitator watching a design review might. Anyone proposing a quality metric for a decision graph owes the reader an admission that the direct thing cannot be measured, only approximated.

The proxy this chapter proposes is a verifiable one, even if it is indirect: how often a decision, once made, gets reversed or substantially revised within days or weeks of being committed. A decision that survives months of contact with real production behavior before anyone touches it again is stronger evidence of a sound original call than one that gets walked back before the sprint that produced it has even ended. The distinction matters because it is exactly the one this chapter must not blur: a decision reversed quickly is different, in kind, from a decision superseded much later because circumstances genuinely changed. Volume IX’s Decision Archives chapter described supersession chains as lineage — decision Z superseding Y superseding X — precisely so that evolving understanding is captured as a healthy sequence rather than treated as a series of failures. A quality proxy that cannot tell a rushed reversal from a healthy, well-earned supersession months later has not measured anything useful; it has just penalized every decision equally for eventually being followed by another one, which describes almost all of them.

Amazon’s own framing sharpens the proxy further. Jeff Bezos’s distinction between decisions that are genuinely irreversible — "one-way doors," which warrant deliberation — and decisions that are cheaply reversible "two-way doors," which do not, implies that a fast reversal means something different depending on which kind of door the team believed it was walking through. Reversing a two-way-door decision inside a week is not a quality failure; it is the door working exactly as designed, the fast, cheap correction that the lightweight process was built to allow. Reversing what the team believed, and should have treated, as a one-way door inside the same week is a much stronger signal that something was wrong the first time. A reversal-rate proxy that ignores which kind of door a decision was framed as is measuring noise. One that reads reversal timing against the decision’s own declared stakes measures something closer to what the word "quality" is actually pointing at.

A decision reversed inside a week is not proof the process was bad. It is the closest smoke alarm this Library can currently propose — and smoke alarms sometimes go off for burnt toast.

Superseded-decision rate

The third proposed metric looks at the decision graph over a window instead of one record at a time: what share of decisions in a project get superseded within, say, a trailing quarter. Read alone, the number is close to uninterpretable. A very low rate could mean genuine stability — a team that reasoned carefully the first time and has had little reason to revisit. It could just as easily mean the opposite: nobody is looking. Volume IX’s Institutional Memory chapter described exactly this failure mode at the level of an entire organization forgetting why a call was made in the first place, and its companion Learning Organizations chapter went further, describing what happens when the structural loop that should reopen a settled decision after new evidence arrives simply does not exist — a finding that never returns to change the artifact it should have updated. A supersession rate near zero, in a system whose underlying assumptions have visibly shifted, looks a great deal like a settled decision nobody has been asked to defend recently, not one that has earned its permanence.

A very high rate carries the same ambiguity in the other direction. It could mean healthy adaptation — a team correctly responding as circumstances change, exactly the case Decision Archives’ own supersession-chain framing was built to support. It could just as easily mean decisions were made carelessly the first time and are now being re-litigated repeatedly, the specific failure Decision Archives named when it warned that teams re-litigate settled questions at full cost when the reasoning behind them cannot be found. Barry Staw’s classic research on escalation of commitment adds a third possibility worth naming: teams personally responsible for a decision’s consequences often become more committed to it after it starts looking wrong, not less — so a suspiciously low rate can also be an organization defending a call it should already have reopened. None of these three explanations — health, carelessness, or defensiveness — is distinguishable from the raw rate alone. The number only becomes useful read alongside what the graph already captures about why each supersession happened, which is the one advantage this metric has that a survey-based version never could.

Three proposed metrics, not three established ones

What this chapter is not claiming

It is worth being blunt about the difference between this chapter and this volume’s Chapter 10, further ahead. DORA’s four keys earned their standing through years of large-sample survey research before anyone treated them as an industry benchmark, and this volume’s opening chapter traces that history in full. Bain’s decision-effectiveness dimensions carry a comparable pedigree, but they were built and validated for organizations answering a survey, not for a queryable graph of timestamped, AI-assisted decisions — a substrate that, as far as this Library can find, nobody has yet applied a decision-quality metric to. What follows here is this Library’s own synthesis of decision-science research that already exists with a technical substrate — the decision graph — that has only recently started to exist, offered as a starting proposal for teams willing to try it against their own history, not as a settled practice with a citation trail behind the specific numbers. Chapter 10 will return directly to exactly the risk this one keeps naming in passing: what happens once a proposed metric like this hardens into a target before anyone has checked whether it is measuring the right thing.