Human Judgment

By 2026, a pull request in a well-instrumented pipeline can clear an eval suite, get scored by an LLM judge, pass every test gate a team has wired into its merge process, and land in production without a single human opening the diff. Nothing in that sentence is hypothetical — this Library’s volume on verification spent eleven chapters showing exactly how each of those layers gets built. What the same volume was honest about, chapter by chapter, is that none of them close the loop completely. An eval is software with its own bugs. A judge is a model with its own blind spots. A test gate checks what someone thought to test for. A change can clear all three and still be wrong in a way none of them were built to catch.

A question this Library left open on purpose

Volume IX’s chapter on engineering wisdom asked what a linter, a rubric, or a decision graph cannot capture, and answered with Michael Polanyi’s distinction between explicit knowledge — what can be written down and handed off intact — and tacit knowledge, which resists exactly that kind of transfer no matter how good the documentation gets. That chapter’s claim was that tacit judgment becomes more valuable, not less, as the codifiable layer around it gets automated. It left one question open on purpose: verification used to be one of the places tacit judgment lived most visibly, in the senior engineer’s felt sense that a diff was fragile before a test could say why. Volume VII then showed that verification itself is now substantially automated too — evals, LLM-as-judge scoring, automated test gates. If judgment used to hide inside the reviewing as much as the coding, and reviewing is now partly a machine’s job, does the category of what stays irreducibly human get smaller, or does it just relocate?

The judge is not a clean substitute

The honest answer starts with what Volume VII’s own chapter on LLM-as-judge already established about the thing doing the judging. An LLM judge is not verification made human-free; it is verification performed by another model, carrying its own documented failure modes into the job — position bias, a preference for longer answers regardless of correctness, and self-preference, a tendency to rate output more highly when it resembles the judge’s own writing, which that chapter reported correlates with how well a model recognizes its own prose rather than with any real increase in accuracy. Add judge drift — a rubric and a judge that scored consistently last quarter producing a visibly different distribution today, with nothing in the process changed except a hosted model version nobody announced — and the picture is not a solved problem wearing a judge’s name. It is largely human judgment’s own failure modes, run through a different substrate at much higher volume.

This reframes the forward-looking question usefully. The codifiable slice of verification — does the diff touch the files it should, does the new code path have a test, does the output satisfy a rubric a careful reviewer could execute the same way twice — is exactly the layer Volume VII showed can be automated well, with real caveats about how carefully it has to be built. What that automation does not touch is the layer underneath it: deciding whether the rubric was asking the right question, noticing that a judge’s confident score sits on a blind spot the rubric never anticipated, recognizing that a change is technically correct and still wrong for reasons no checklist named. By Polanyi’s own definition, that layer cannot be handed to an automated system, because tacit knowledge is precisely the kind of knowing that resists being made explicit enough to encode. Volume IX made that argument for the coding half of the pipeline. Nothing in Volume VII’s own honest accounting of eval, judge, and gate limitations gives a reason to think the reviewing half is different.

The open question is not technical

Which points to the question this chapter actually needs to sit with, and it is not a capability question at all. Volume XI’s chapter on skill formation described a genuinely contested debate: whether the engineers who would develop this kind of judgment are still getting the chances that used to build it, now that an agent does the drafting and the struggling both. The Anthropic trial that chapter examined found junior engineers who delegated to an assistant scoring measurably lower on a later comprehension quiz than engineers who worked unassisted — a finding that chapter was careful to call real but narrow, not a verdict. DORA’s expertise paradox names a compatible risk from the organizational side: AI can close a knowledge gap fast enough to skip the struggle that builds real expertise, producing what it bluntly calls false expertise. Both matter here because Volume IX already established that tacit judgment only moves person to person, through something close to apprenticeship — watching an expert work, live or replayed, until a pattern repeats often enough to be recognized on sight. If fewer engineers get the repetitions that build that recognition, the category of what stays human does not shrink because it stopped mattering. It shrinks because fewer people are left who can supply it.

This Library’s own synthesis, hedged

Put together, the chapters this one draws on point somewhere more specific than “AI will keep automating more of the pipeline.” On the evidence this Library has gathered across its first eleven volumes — offered here as this Library’s own synthesis, not a settled fact about a future nobody has yet observed — human judgment’s role does not look like it is shrinking. It looks like it is concentrating: into fewer moments, each carrying more weight. The initial framing of what problem is actually worth solving, before any eval or rubric exists to check the answer against. The decision to trust or distrust an output that cleared every automated gate anyway. The recognition — Polanyi’s felt sense, arriving before it can be fully articulated — that something is subtly wrong in a place none of the checks were built to look. Those three moments do not get smaller as automation improves at everything around them. If anything they get harder to staff, because the apprenticeship that used to produce people capable of them was never designed on purpose. It happened by accident, inside work that increasingly does not happen anymore.