The Operon Library

Volume VII · Generation is cheap. Verification is the job.

Verification & Trust

Generation cost collapsed while verification didn't — the defining asymmetry of the era, and the second-biggest gap in the first edition. The volume covers evals as engineering practice, LLM-as-judge, adversarial verification, benchmark skepticism, test and merge gates, provenance, and agent security — a topic absent from every volume in v1 — closing by treating verification as a first-class architectural layer, not a phase.

Chapters
11
Published
11
01The Verification AsymmetryGeneration cost collapsed; verification didn't — the economic and cognitive asymmetry that defines the era.8 min readPublished
02Evals as an Engineering PracticeFrom vibes to behavioral test suites for agents; regression evals for workflows; evals-in-the-loop.8 min readPublished
03LLM-as-JudgeThe craft and the failure modes: rubric design, single-call scoring, judge drift, self-preference bias.8 min readPublished
04Adversarial VerificationRefuter panels, multi-lens review (correctness/security/perf), kill-if-majority-refutes.9 min readPublished
05Benchmark SkepticismThe SWE-bench Illusion — memorization and contamination — and why "X% on SWE-bench" is not an economics input.8 min readPublished
06Test Gates & Merge GatesPolicy-enforced verification: test-gated merges, required checks, gate design that doesn't rot.9 min readPublished
07The New Code ReviewReview economics under AI volume: what human review is for now, and reviewing AI code with AI assistance.7 min readPublished
08Provenance & AttributionWho (or what) wrote this line: agent-authored change tracking, audit trails, attribution in incident review.7 min readPublished
09Agent SecurityPrompt injection, data exfiltration, supply-chain attacks like hallucinated-package squatting, and injection-resistant approval architectures.8 min readPublished
10Trust CalibrationWhen to trust, spot-check, or re-derive; confidence scoring; automation bias and its inverse.9 min readPublished
11The Verification LayerCapstone — verification as a first-class architectural layer, not a phase.5 min readPublished

Key sources: SWE-Bench Illusion paper + benchmark-audit wave · Anthropic LLM-as-judge findings · Faros review-time data · OWASP LLM Top 10 · OpenTelemetry GenAI conventions (for provenance plumbing)

Related