Systems Before Models

A VP of engineering reads a vendor benchmark update — the coding agent her teams have used for a year just jumped several points on a well-known leaderboard — and approves the org-wide upgrade in the same meeting. No pilot, no rollback plan beyond the usual one: if something breaks, revert. The reasoning is not unsound. A better underlying model should mean better output, and the upgrade costs nothing but a config change. Three months later, the metrics she reports upward look almost identical to the quarter before. Pull requests opened per engineer are up. Pull requests merged per week is flat. The median time a change spends waiting in review has gotten longer, not shorter.

Nothing about the story is unusual, and nothing about it is really about the model. The team asked a model-level question — which coding agent scores highest — and got a model-level answer. The organization that receives the model’s output, however, is not a model-level system. It is a network of people, review queues, test suites, incident channels, and incentives that decides, collectively and mostly implicitly, how much of what the model produces actually reaches production. Swapping the model changes one node in that network. It does not touch the network.

This Library’s Volume VII, in a chapter called Benchmark Skepticism, already made the narrower version of this argument: a leaderboard percentage is a benchmark score, not a business metric, and picking a tool by its highest number substitutes a model-level measurement for a team-level question. The point here generalizes one level up. Even a model chosen carefully, on real evidence, on a team’s own repositories, is still only one component of the system that determines whether the organization gets more value out of it. The model can be exactly right and the outcome can still fail to move — because the constraint was never the model.

What the dashboard actually shows

The pattern generalizes past any one VP’s anecdote. Google’s 2025 DORA report, surveying thousands of engineering professionals on AI-assisted development, found that AI’s primary organizational effect is not improvement but amplification — it magnifies whatever a team already does well, and just as reliably magnifies whatever it does badly. A separate ten-thousand-developer dataset examined by Faros AI put a number on the mechanism: teams using AI completed 21% more tasks and nearly doubled pull-request throughput, while delivery metrics stayed essentially flat and review time grew 91%. Generation got faster. The system’s actual output did not, because generation was never the constraint. Review capacity was, and nobody redesigned the review process before pointing a faster generator at it.

The vocabulary problem

Most engineers who have lived through a story like the VP’s can feel that something is off — more code moving, less of it actually shipping, a reviewer who used to argue about architecture now just trying to keep the queue under control. What they usually lack is a name for what happened, and a way to reason about it that goes deeper than “our process needs work.” Systems thinking is the discipline built to supply both, and it did not originate in software. Donella Meadows spent much of her career studying why interventions in complex systems — ecosystems, economies, organizations — so often produce the opposite of what their designers intended, work she distilled into Thinking in Systems: A Primer. Peter Senge spent his studying why organizations specifically keep failing to solve the same problems, work he published as The Fifth Discipline: The Art and Practice of the Learning Organization. Neither book has anything to say about large language models. Both have a great deal to say about why a smarter model, dropped into an unexamined system, tends to reproduce the system’s existing failure mode faster rather than fix it.

A system, defined

Meadows’s definition is deliberately plain: a system is a set of elements, interconnected in a way that produces a pattern of behavior, organized around some purpose or function. None of the three parts is optional, and the middle one does the real work. Two teams can have identical elements — the same model, the same headcount, the same tools — and produce completely different outcomes, because their interconnections differ: how fast a bug report reaches the person who can fix it, whether a failing test blocks a merge or gets skipped under deadline pressure, whether a disagreement about approach gets resolved in an hour or festers for a sprint. Behavior is a property of the structure, not of any element sitting inside it. This is the sentence worth sitting with before the next model evaluation: an AI-assisted engineering organization producing bad outcomes — shipped bugs, a stalled review queue, a team quietly burning out — very rarely has a model problem at its root. It has a structure problem the model is currently operating inside of, visible or not.

Meadows makes one further point that engineering leaders tend to resist the first time they hear it: a system’s real purpose is not what its charter says, it is whatever the system’s actual behavior reveals it to be. A code review process whose stated purpose is “ship quality software quickly,” but whose actual behavior is a three-week queue nobody has fixed in a year, does not have a broken purpose — it has revealed its real one, which might be something closer to “protect the codebase from the volume of change now arriving,” whether any single person designed it to do that or not. This is not a cynical reading. It is a diagnostic one: watch what a system does under load, not what its mission statement claims, and the actual leverage points become visible.

A system’s mission statement describes what it is supposed to do. A system under load reveals what it is actually built to do — and those are frequently two different systems.

On Donella Meadows’s account of system purpose

Shallow fixes, deep structure

Peter Senge’s contribution sharpens the mechanism further. One of the recurring patterns he catalogs in The Fifth Discipline — an organization reaching for a symptomatic fix instead of a fundamental one — describes almost exactly what a benchmark-driven model swap tends to be. The symptomatic fix is fast, visible, and requires no one to confront an uncomfortable structural question. The fundamental fix is slower, requires organizational will, and usually means someone has to own redesigning a process rather than approving a vendor change. Senge’s warning is that the symptomatic fix does not just fail to solve the underlying problem — it can erode the organization’s capacity to apply the fundamental fix later, because the symptom becomes bearable enough that the urgency to do the harder work quietly disappears. A faster model that produces more code for the same understaffed review queue does not just fail to fix the bottleneck. It can make the bottleneck less visible in the short term while making it worse in practice, because the volume of change waiting behind it keeps growing.

Leverage points for an AI-assisted engineering system

Meadows is best known for a related idea worth borrowing directly: leverage points, the places in a system where a small intervention produces an outsized effect, ranked from shallow — easy to change, weak effect — to deep — hard to change, transformative effect. Her full hierarchy runs to a dozen tiers, developed at length in the 1999 essay Leverage Points: Places to Intervene in a System and carried into the book. A simplified three-tier version is enough to place a model swap correctly against the alternatives an engineering leader actually controls.

Placed on this ladder, the VP’s upgrade sits on the bottom rung. It was worth doing — a better model is not worthless — but it was never going to move a bottleneck sitting two rungs up. The teams that get a durable return from a model upgrade are, overwhelmingly, teams that made the structural move first: they shortened the feedback loop between a change and its review, they decided explicitly what a test gate is actually for, they gave someone real authority to say a queue is broken before it became invisible. The model upgrade then amplifies a structure that was already working — which is exactly the shape of the DORA finding this chapter opened with.

What a systems-level dashboard would track

Operon has not run a controlled study isolating model swaps from structural changes, so the figures below are illustrative — the shape of the dashboard a team should build before crediting or blaming a model, not a measured result. The point of instrumenting at this level is to make the amplifier effect visible before it compounds for a quarter: log the model or tool change as an event on the same timeline as merge rate, review queue depth, and defect escape rate, and the question “did the upgrade help” gets an answer instead of a feeling.

The volume ahead

This chapter is the frame the rest of this volume works inside. The chapters immediately ahead look at the loops that actually govern day-to-day engineering work — how fast feedback closes, and the everyday build-test-review-deploy cycle an AI-assisted team actually lives inside — then turn to how small decisions compound, how complexity gets managed rather than eliminated, how information actually flows through an organization, who has the standing to decide what, and how the organization itself behaves as a system with its own delays and failure modes. Chapter 9 returns explicitly to AI-assisted engineering once that toolkit is assembled; the volume closes with a capstone meant to be worked through against a reader’s own organization, not read passively. None of it argues against a good model. It argues that a good model is one component of a system whose structure was set long before the model arrived, and will still be there long after this one is replaced by a better one.