Predictive Engineering

A team lead sizes up a proposed session the way team leads have always done: reading the goal, picturing how similar work has gone before, arriving at a number to put in the sprint plan. “This one’s a $340, forty-minute job,” someone says, and the figure goes into the plan as though it had been measured rather than guessed. Nobody asks how many past jobs that guess is actually standing on, because until recently there was rarely a real answer — most organizations had no session-level history to stand on at all.

That has started to change. The instrumentation this volume has spent its earlier chapters building — cost and duration captured per session, outcomes tagged as merged, reverted, or abandoned, patterns surfaced across sessions and across a team — is, almost as a side effect, a genuine historical record of how similar work has actually gone. An organization that has been logging its AI-assisted sessions for even a few months has something a project manager estimating a bridge or a software release has rarely had this cheaply: a reference class of comparable past work, sitting in a database rather than in anyone’s memory.

The inside view, and what it costs

The instinct the team lead used above has a name, and the name comes with decades of unflattering evidence behind it. Decision researchers call it the inside view: reasoning about a new task by examining its specific features — the files it touches, the unfamiliar corner of the codebase, the person who will do the work — and building an estimate up from that examination alone. It feels rigorous, because it genuinely is reasoning about the task in front of someone rather than waving at history. It is also, reliably, wrong in the same direction: too fast, too cheap, too clean. A review spanning multiple companies and study designs found that somewhere between sixty and eighty percent of software projects overrun their own effort or schedule estimates — not because the people estimating were careless, but because the inside view structurally omits an entire category of problem: the things that do not go the way the plan assumed.

The outside view

Daniel Kahneman and Dan Lovallo gave the corrective a name in a 1993 paper that has since become foundational to forecasting research generally: the outside view. Instead of reasoning about how a specific new task will go, ask how tasks that looked like this one actually went, across as large and honest a sample as can be assembled — build a probability distribution from that reference class, then place the new task inside it rather than reasoning about it in isolation. The paper’s deeper finding is worth sitting with. Decision-makers, the authors argued, tend to be simultaneously too optimistic about individual projects and too cautious about portfolios of them — bold on the forecast, timid on the bet — and both errors trace back to the same habit: treating each case as unique instead of asking what the numbers already know about cases like it.

Bent Flyvbjerg spent much of the following two decades turning that idea into applied practice, at a scale where the stakes made the extra friction worth it: rail lines, tunnels, national transport programs. His account of the method’s early adoption — cost forecasts for projects like Edinburgh’s tram network and London’s Crossrail, benchmarked against the real outcomes of comparable infrastructure rather than the optimistic case a project’s own planners had built from scratch — became the basis for a UK Department for Transport guidance document in 2004, precisely because the naive, bottom-up estimate had been shown, project after project, to be the less trustworthy one. Reference-class forecasting is not a hypothesis. It has a track record, built one large, expensive, well-documented failure at a time.

A reference class at session grain

What is new, and genuinely untested at this scale, is running the same corrective on a far smaller and faster unit of work. Flyvbjerg’s reference classes were built from tens or hundreds of infrastructure megaprojects tracked over years. An organization instrumenting AI-assisted sessions can, in principle, assemble a reference class from hundreds of comparable sessions inside a single quarter — exactly the “similar past work, what did it cost, how long did it take, did it succeed” query the cross-session and team-level intelligence this volume has already covered exists to answer. That is a real advantage: the data accumulates fast and is cheap to query. It is not a reason to borrow more confidence than the sample actually supports. A method proven on bridges has not yet been stress-tested on units of work this small, this fast, and this variable, and the honest position in mid-2026 is that AI-engineering-specific predictive practice is young. What exists are the raw ingredients — session records, cost data, outcome tags — and a well-proven general method for using them. What does not yet exist is a mature, settled literature on how well that method performs at this particular grain.

Even inside a single session the ground is less solid than it looks. A 2026 study that asked frontier models to predict their own token usage before finishing a coding task found only a weak-to-moderate relationship between a model’s estimate and what it actually spent — a correlation reaching at most 0.39 — with models systematically underestimating the real cost. If the system doing the work cannot forecast its own near-term spend reliably, a reference class built across many such sessions inherits that noise at the input layer, before any statistical aggregation across sessions even begins.

What a forecast owes the room

None of this argues against forecasting. It argues for treating a forecast as something that carries its own credentials, rather than a bare figure delivered with identical confidence whether it rests on three sessions or three hundred. A defensible prediction states three things a bare number omits by default: a range instead of a point, an explicit account of the reference class behind it, and an honest list of what the underlying model can and cannot see.

What the telemetry should show

The forecast that changes its own outcome

There is a complication worth naming plainly rather than stepping around: a forecast in this domain is not a neutral description of a future that will happen regardless of being spoken aloud. The sociologist Robert Merton described the general shape of the problem in 1948 — a public prediction becomes, in his account, a new element in the very situation it describes, capable of making itself true or of making itself false depending on how people react to hearing it. A session a reference-class forecast flags as expensive or risky is, in a harness with any kind of gate at all, exactly the session most likely to get extra scrutiny — the checkpoint pauses this Library’s account of checkpoint thinking argues for, an engineer reading the diff more carefully, a plan reviewed twice instead of once. That scrutiny is the correct response to a risk signal. It is also an intervention that changes the very outcome the forecast was trying to predict, so a session that would have cost $500 unattended might come in at $350 specifically because the forecast warned someone to watch it — which means, over time, “the forecast was accurate” and “the forecast caused its own accuracy” become genuinely difficult to pull apart from the outside.

A public prediction becomes a new element in the situation it describes.

Robert Merton, on self-fulfilling and self-defeating prophecy, 1948

None of this is a reason to distrust forecasting, or to stop checkpointing high-risk work — both practices remain correct on their own terms. It is a reason to track forecasts against outcomes with some humility about what “the forecast was right” actually proves, and to resist treating a run of confirmed predictions as evidence the model is capturing more than it is. Sometimes it is only capturing that people act differently once they have been warned.

A forecast delivered as a single confident number is, in miniature, exactly the kind of metric this volume’s next chapter spends its length warning about: one that looks more rigorous than the evidence behind it, and that changes behavior in ways nobody accounted for when they built it. The discipline this chapter argues for — state the range, state the sample, state what the model cannot see — is not a separate concern from that one. It is the same concern, applied to a number about the future instead of a number about the past.