Team & Organizational Intelligence

A team lead at a hundred-engineer product organization asks a reasonable question three months into a company-wide AI-agent rollout: which teams are actually getting value from this, and which are lagging? Someone on the platform team already has the data — every session, every team, every dollar, every merge outcome, sitting in exactly the telemetry this volume has spent seven chapters arguing an organization should collect. The natural next step, taken almost as an afterthought, is a spreadsheet: engineers down the rows, average session cost and merge rate across the columns, sorted worst to best. It circulates in a Slack channel within a week. Nobody asked the platform team to build a performance ranking. It built one anyway, because the columns were already there and sorting takes one click.

This is not a mistake unique to AI tooling. It is the oldest failure mode in engineering measurement, arriving in a column now labeled tokens instead of one once labeled lines of code or story points closed. What earns it a full chapter here, rather than a footnote, is the shape of the temptation: the identical dataset that makes genuinely useful cross-team intelligence possible — which categories of work are expensive for every team, not just one; which team has quietly solved a problem three others are still paying for — is, row for row, the same dataset that produces the leaderboard. Nothing in the schema tells you which use you are building. Only the aggregation does.

What the cross-session view can actually see

The previous chapter in this volume treats a single AI session — one goal, one engineer, one sitting — as the smallest unit worth analyzing on its own: cost, outcome, duration. Widen the aperture across hundreds or thousands of those sessions, spanning many engineers and many teams over months, and patterns appear that no single team’s own history is large enough to see reliably. A task category — database migration work, say, or anything touching a particular legacy service — might turn out expensive and error-prone everywhere it appears, across every team that has touched it, which is a fact about the work, not about any one team’s competence. A different task category might be expensive for exactly one team and routine for three others working in adjacent parts of the same codebase, which is a different fact entirely: a knowledge-sharing gap, an approach one team never learned that another team already has.

Telling those two cases apart — hard everywhere, versus hard only here — is close to impossible from inside a single team’s own session history. A team of six or eight engineers running a few dozen sessions a month against one category of work does not generate enough samples to separate signal from noise; it generates a story someone tells at retro. It is exactly the kind of question a larger, pooled dataset answers well and a small one answers only by guessing. That is the genuine case for cross-team analysis, and nothing about it requires identifying a single engineer anywhere in the pipeline.

The same rows, two different products

The trouble is that the query answering the honest question and the query producing the leaderboard read from the same table. Group by task category and time period, and the output is org-wide intelligence: durable, actionable, safe to publish widely. Group by engineer instead — or hold the group-by constant but drill down far enough, one team, one week, one category — and the same rows resolve to a set small enough that anyone who knows the team’s roster can reverse-engineer who they belong to. No name column is required for a metric to become a surveillance tool. A team of two working one category of task for one week is a de facto individual scorecard wearing a team label.

A team-level metric with a small enough team is not a team-level metric. It is an individual metric with a group photo attached.

A borrowed discipline: measure the system, not the person

Mature reliability engineering practice solved a structurally similar problem more than a decade ago, and this Library has already drawn on it once, in Volume IX’s chapter on Learning Organizations. John Allspaw’s 2012 argument for blameless postmortems at Etsy holds that engineers who fear individual blame withhold exactly the detail an incident investigation needs, and that treating a failure as a symptom of a system’s vulnerabilities rather than one person’s mistake is the precondition for getting an honest account at all. Google’s own Site Reliability Engineering practice, formalized a few years later, builds postmortem review — including the aggregate analysis of many postmortems together — around the same discipline: the system failed, not an engineer, and the record is written and read that way on purpose.

The parallel to team- and org-level AI engineering intelligence is direct, not decorative. A blameless postmortem culture does not refuse to look at data — Google reviews postmortems in aggregate precisely because patterns across many incidents reveal things no single incident does, the same argument this chapter is making for task-category benchmarking across many sessions. What it refuses is analysis at a grain fine enough to land on one person’s name. The discipline is not “collect less data.” It is “aggregate at a level the data structurally cannot be walked back down to an individual from,” applied as consistently to session telemetry as it already is to incident data in any SRE organization worth the name.

What DORA and SPACE already say about this

DORA’s own guidance on its four keys — the closest thing the DevOps metrics community has to a settled canon, and a framework this volume engages directly elsewhere — is unambiguous on the point: software delivery performance, in DORA’s own words, “is not an individual measure; it measures your ability to change and update an application, and this can only be done by teams.” The same guidance warns that isolating teams against each other with head-to-head metrics produces friction and finger-pointing rather than improvement — the team-versus-team version of the exact failure this chapter opened with at the individual level. Neither warning is a hedge added for legal comfort. It is a description, from the people who designed the metrics, of what they were built to answer and what they were never designed to answer.

The academic case predates DORA’s current guidance and comes from a different direction. The SPACE framework — Forsgren, Storey, Maddila, Zimmermann, Houck, and Butler, writing in ACM Queue in 2021 — argues that developer productivity “cannot be measured by a single metric or dimension,” precisely because any single number, applied to one person, invites the gaming and morale damage this chapter is describing at team scale. DORA’s newer guidance on AI-era metrics extends the same warning to token-based measurement specifically: a 2026 DORA piece on what it calls “tokenmaxxing” describes engineers running agents on throwaway work purely to keep a personal token count above the office average, and teams withholding their most effective prompts and workflows from each other to stay ahead in a competition nobody asked them to enter. That is Goodhart’s law arriving in an AI-specific costume it has not worn before — but it is the same law.

Getting Goodhart’s law right

The law is worth citing precisely, because it is one of the more reliably misquoted ideas in the metrics literature. The version everyone repeats — “when a measure becomes a target, it ceases to be a good measure” — is not the economist Charles Goodhart’s own phrasing. Goodhart, writing on UK monetary policy in 1975, stated the idea in denser form: any observed statistical regularity tends to collapse once pressure is placed on it for control purposes. The pithier version in circulation today comes from the anthropologist Marilyn Strathern, who restated it in 1997 while writing about audit culture in British universities — generalizing Goodhart’s monetary-policy point into a claim about human institutions broadly, which is the form the software industry actually uses. Both are correct, and it matters which one is being invoked: Goodhart’s original describes what happens to a statistic once it is targeted; Strathern’s describes what happens to people once they know the statistic is watching them. A team-level session leaderboard is squarely Strathern’s problem, not Goodhart’s original one.

An aggregation principle, stated plainly

None of this argues against building team- and org-level AI engineering intelligence. It argues for a specific, checkable discipline in how it gets built: aggregate at a level — team, task category, time period — coarse enough that the output cannot be walked back down to one person’s name, and decline to publish any cut of the data fine enough to fail that test, no matter how the request is framed. This is the same aggregation discipline mature engineering organizations already apply to postmortem findings (system-level, not individual-blame) and to code review metrics (review-quality trends across a team, not a per-reviewer scorecard). It is not a new idea invented for AI tooling — it is an old one that AI tooling’s session-level granularity makes newly tempting to skip, because the data now arrives already itemized by person and timestamp, whether anyone asked for it that way or not.

What this actually buys an organization, done honestly

Done at the right grain, the analysis earns its keep in ways a leaderboard never could. An org-wide difficulty map — the same kind of cross-session aggregation useful at the individual level, run instead across every team touching a given category of work — tells a platform group where to invest in tooling or documentation, because the expense shows up everywhere rather than in one team’s numbers alone. A knowledge-sharing surface built the same way answers a quieter, more useful question: which team has already found a cheap, reliable approach to a task category that is expensive for three others, and is there a fifteen-minute conversation that saves those three teams months of relearning it independently. Neither output requires knowing which engineer typed which prompt. Both require knowing which team and which kind of work, at a grain coarse enough that the data cannot be un-aggregated by anyone determined to try.

An honest limitation

It is worth saying plainly what this chapter is and is not built from. Almost none of the material grounding it is AI-engineering-specific research, because almost none yet exists at the maturity of SRE postmortem culture or the DORA and SPACE literature on DevOps metrics — both of which took the better part of a decade of publication, replication, and industry argument to reach the settled form cited here. What this chapter offers instead is a transplant: a discipline built for incident data and code-review data, applied to session telemetry because the underlying problem — aggregate carelessly, and a system built to find genuine patterns becomes a tool for finding people to blame — does not appear to be specific to any one kind of engineering data. That transplant is a reasonable bet, not a proven one. An organization building team-level AI intelligence today is doing something closer to applying a mature discipline in an unfamiliar setting than following an established playbook written for this exact problem. The next chapter in this volume takes up Goodhart’s law directly as its organizing idea; this chapter has met it here first, in the one place aggregated engineering data does the most damage when it is allowed to slip below the level of a person.