Trust Calibration

A senior engineer six months into a new AI harness describes the arc this way: the first month, she read every generated diff line by line, treating the tool the way she would treat a promising but unproven hire. By the second month she was skimming. By the third, she was opening maybe one diff in five, waving the rest through on the strength of a habit that had quietly formed and never once been examined. Nothing forced the change. No policy shifted, no incident occurred, nobody told her the tool had gotten better. Ask her today whether her current level of scrutiny is correct, and she cannot answer — because she never decided on it. It simply wore in.

This drift has a name in a field far older than AI-assisted coding, and it runs in both directions at once. Automation bias is the well-documented tendency to over-rely on an automated system — to defer to its output even when a person’s own training, or the evidence directly in front of them, says otherwise. Algorithm aversion is the mirror image: under-trusting a system that is, by any measured standard, more reliable than the human judgment being substituted for it. Both are real, both are decades deep in the human-factors literature, and — this is the part most confidence-score dashboards miss entirely — both can happen to the same team, about the same tool, in the same quarter, at different moments in the adoption curve.

Trusting too much

The canonical demonstration comes from aviation. In a series of simulated flight tasks, Kathleen Mosier, Linda Skitka and colleagues gave pilots a highly — but not perfectly — reliable automated monitoring aid and compared their performance against pilots working without it. The automated group performed worse. They made errors of omission, missing events the aid failed to flag even when other valid indicators were plainly available, and errors of commission, following the aid’s recommendation even when it directly contradicted their own training and instruments that were still showing correct information. The aid was right almost all of the time. Almost all of the time turned out to be enough to make the exceptions invisible.

The pattern generalizes well past cockpits. A systematic review of automation bias in clinical decision-support systems — the kind of tool that flags a likely diagnosis or drug interaction — found that when a decision aid gave erroneous advice, clinicians using it were meaningfully more likely to make an incorrect call than clinicians working without it, with erroneous advice raising the odds of a wrong decision by roughly a quarter. The same review surfaced a genuinely useful design lesson: systems that attached a confidence level to each individual piece of advice, rather than asserting one fixed overall reliability rating for the whole tool, produced better-calibrated reliance than systems that stated their trustworthiness once and left it there. A static score invites a static level of trust. A score that updates per decision does not.

Trusting too little

The opposite failure is just as well documented and, until recently, far less discussed in engineering contexts. Berkeley Dietvorst, Joseph Simmons and Cade Massey ran a series of forecasting experiments in which participants watched an algorithm make predictions, a human make predictions, or both, then chose whose future predictions to stake their own incentives on. People lost confidence in the algorithm faster than they lost confidence in the human — even after watching the algorithm outperform the human, and even when the algorithm’s error was no larger than the kind of error a person makes constantly without anyone revising their opinion of human judgment downward. Seeing an algorithm be wrong even once was enough to make people prefer a demonstrably worse forecaster, provided that forecaster was a person.

Whether this generalizes cleanly to every context is itself contested. A separate line of research by Jennifer Logg, Julia Minson and Don Moore found the reverse effect under different conditions — ordinary people, given a choice between an algorithm’s estimate and a stranger’s, preferred the algorithm, an effect the researchers called algorithm appreciation. It weakened sharply once participants had real expertise in the task, or were choosing between the algorithm and their own judgment rather than someone else’s — which is precisely the situation most engineers are actually in when reviewing an AI-generated diff. The two literatures agree on the mechanism even where they disagree on the direction: trust in an automated system is fragile and unevenly distributed, tipping hard toward over-reliance or under-reliance depending on who is being asked and what they stand to give up. One mitigation replicates across both traditions: letting a person retain even a small amount of control — the ability to slightly adjust an algorithm’s output rather than only accept or reject it wholesale — restored willingness to use an imperfect system, and the resulting decisions measurably beat both unaided human judgment and the unmodified algorithm.

The two failure modes are not opposites that cancel out. A team can be over-trusting one part of its harness and under-trusting another, in the same week, without noticing either.

On the coexistence of automation bias and algorithm aversion

The bias one level up

None of this is solved by adding a confidence score to the harness — though a well-built score is a real improvement over the alternative. John Lee and Katrina See’s foundational review of trust in automation frames the goal as calibration: trust that tracks a system’s actual, demonstrated reliability rather than sitting fixed at either extreme regardless of what the evidence says. This volume’s workflow-engineering counterpart, the Checkpoint Thinking chapter, already did the harder half of that work — it specified what a defensible confidence score should be computed from: file count, dependency reach, scope-boundary crossings, context utilization, and above all a track record built from real prior outcomes rather than a model narrating its own certainty. That is the what. This chapter is about what happens next, because a good score does not enforce its own correct use.

A score is still just one more piece of information a person has to decide how much to weigh — and the same two failure modes that govern trust in an AI’s output govern trust in the score about that output, one level removed. A team that starts treating any checkpoint above a threshold as automatically safe has replayed automation bias against its own instrumentation rather than against the model. A team that keeps reading every diff in full regardless of what the score says has replayed algorithm aversion against a tool it built for the specific purpose of being trusted when it has earned trust, burning the entire cost of building the score for nothing. Neither failure requires the score itself to be wrong.

And a bare gut feeling is a poor tiebreaker at either level. This Library has already established, in the same checkpoint-thinking discussion, that a model’s self-report of its own confidence is close to worthless — a 2026 study found agents that actually succeeded at their task only 22% of the time predicting a 77% success rate for themselves, with pre-execution guesses, made with strictly less information, often out-discriminating the model’s own after-the-fact review of its work. The human side fares little better on its own: the reason METR’s randomized trial keeps getting cited across this Library is that experienced developers measured 19% slower with early AI tools estimated they had been about 20% faster, a forty-point gap between what happened and what it felt like. If neither the model’s self-report nor an unaided human hunch is trustworthy, the only input worth building a decision around is a score constructed from outcomes external to both — read by a person trained to weigh it the way it was built to be weighed, not the way habit happens to weigh it that week.

A three-way decision

None of this resolves into one rule, because the correct level of trust is a function of stakes and reversibility, not a fixed property of the tool. The practical question a reviewer actually faces, dozens of times a day, is which of three postures a given piece of AI output deserves: accept it outright, spend a few minutes checking one specific thing, or set the output aside and work through the problem independently. The table below is not a substitute for judgment — nothing in the research above suggests a fixed rule would survive contact with a real codebase — but it names the inputs that should be driving the choice, instead of leaving the decision to whichever bias happens to be dominant that week.

The middle row does the most work and gets the least attention. A team that only ever fully trusts or fully re-derives is running a binary decision with a stakes threshold buried in the middle of it, which is a coarser instrument than most situations call for. The entire value of a confidence score built from real signals is that it can point at the one thing worth checking in an otherwise trustworthy step, rather than forcing a reviewer to read everything or read nothing.

What a trust-decision log would need to show

Recalibrating deliberately

Two mechanisms, both borrowed directly from the automation-bias literature rather than invented for AI coding, are worth building into a review habit rather than leaving to individual willpower. The first is accountability. Studies going back to the original aviation work found that pilots who expected to have to justify a decision, before making it, verified the automated aid’s output significantly more often and committed fewer automation-related errors than pilots facing no such expectation. The translation to a code review habit is almost literal: a checkpoint approval that requires one written line naming the specific thing the reviewer checked produces different behavior than a checkpoint approval that is just a button, even when the button is technically sufficient to unblock the session.

The second is keeping the interaction with the automation real rather than binary. The finding that people stay engaged with an imperfect algorithm when they can adjust its output, even slightly, rather than only accept or reject it wholesale, suggests that a review interface offering nothing between approve and revert-everything is quietly training reviewers toward one of the two failure modes — probably automation bias, since editing takes more effort than clicking approve. Underneath both mechanisms sits the only durable fix for calibration drift: logging, for every trust decision a team actually makes, which posture was chosen and what happened afterward, then periodically checking whether the two agree. A team that never checks this has no way of knowing which direction its trust has drifted — only a vague sense, probably wrong by the weight of every study cited above, that things feel about right.

None of this is a one-time setup. The engineer who went from reading every diff to reading one in five did not make a decision; she absorbed a habit, the way anyone absorbs a habit around a tool that is right often enough to stop being interesting. Trust calibration is the discipline of noticing that drift before an incident notices it first — checking, on some regular cadence, whether the posture a team defaults to still matches what its own outcomes say it should be. The next chapter treats verification as something a system is built with, not something a person remembers to do. This one is about the person still deciding, checkpoint by checkpoint, how much of that system to actually believe.