Agents review everything, humans review red

We spend human review attention uniformly across every pull request, as if all changes were equally dangerous. That is a fixable bug, and fixing it starts from one price: a human produces tokens about a thousand times more expensively than a model does (the arithmetic is in Human Tokens). This piece is about spending the expensive token only where it pays.

Humans fail too, at measured rates

The comparison people make implicitly is one agent pass against an idealized human who doesn't err. The measured human looks like this:

• The figures McConnell compiled put delivered code anywhere from 15 to 50 defects per thousand lines, depending on whose average you take (old numbers, still directionally sound).
• A normal code review catches maybe 20 to 35 percent of defects. Formal inspection, the kind with a moderator and a checklist and a scheduled meeting, gets to 55 or 60. Same book.

I use those numbers for exactly one purpose: establishing that human review is measurably leaky. They describe humans reviewing human-authored code, which has a different defect shape than agent-authored code, where you see plausible-but-wrong logic and the occasional invented API instead of typos. They can't calibrate agent review, and anyone extending them that way, including me in an earlier draft, is borrowing authority rather than citing evidence.

What most teams actually buy is one review pass, because a second reviewer doubles an expensive line item. One draft by a fatigable author, one four-minute skim ending in LGTM, ship. None of that is negligence. The price of human attention set the ceiling.

Agent passes price differently. An hour of engineer time is $100, which buys millions of output tokens, which is a lot of review over a feature-sized diff. Two structural properties come along free, and neither one needs a formula:

• No vigilance decay. The peer-review studies put the human cliff at about an hour and four hundred lines, so a thorough pass on a big diff is worst in the long boring middle, which is where the interesting bug is. The fortieth agent pass arrives as fresh as the first.
• No ego. The reviewing agent didn't write the code and isn't defending it in standup tomorrow.

The formula everyone reaches for, (1−p)ⁿ, a 50 percent catch rate over six passes leaving a 1.6 percent miss, is arithmetic dressed as evidence. The only calibrated p available is the human one, from the human literature. The closest thing on the model side, OpenAI's critic experiments, measured catch rates on bugs planted for the test, which is a benchmark, not a calibration. The multiplier I'd love to quote you does not exist. The direction holds and it's enough to act on: redundancy is nearly free, so buy a lot of it. The magnitude stays unclaimed until someone measures it, which is the last section of this piece.

Two kinds of defect, and only one yields to redundancy

Passes from one model family don't fail independently. They share blind spots, which is a worse problem than sharing noise, because noise averages out and a blind spot doesn't.

• Execution defects. Wrong logic, missed edge case, broken test, bad escaping. A model can plausibly catch these, and partial independence can be engineered: separate lenses per pass (correctness, security, "prove this breaks"), fresh context each time, and tests, which are a check that isn't a model at all. Redundancy works here.
• Intent defects. The code faithfully implements a misreading of what the business wanted, or misses a constraint that lived in one person's head and was never written down. For anything that never made it into text a reviewer can read, the catch rate isn't fifty percent and correlated, it's zero by construction, and no number of passes moves it — you can't find what isn't in your inputs. These are disproportionately the expensive failures, the ones where the code did exactly what it was told.

Human review isn't obsolete, it's mispriced. Spent uniformly, it goes mostly to execution defects the agents already caught, and starves the intent defects only a human can catch.

There's a cost people skip when they hear "six passes": somebody has to adjudicate. Six lenses produce six lists with overlapping and sometimes contradictory findings, and reconciling those is judgment work billed at the human rate. Left unbudgeted, cheap redundancy turns into an expensive inbox, and the redundancy argument quietly eats itself. Two things stop that. Passes emit structured verdicts, one block per lens with finding, severity, and confidence, so most of the agreement collapses cheaply, exact matches dedupe themselves, and only the residue needs judgment. And the risk tier decides whether a human adjudicates at all: on a low-risk change the gate does it, and nobody reads the findings unless something crosses a threshold.

Which is the proposal.

Red, yellow, green

Two axes engineering orgs already trust, because they run this exact logic for deploys: reversibility (can we roll it back cheaply?) times blast radius (how many users and systems feel it?). One-way doors and two-way doors, pointed at review attention instead of at releases.

Agents review everything, humans review red.

The rules that make it survive a skeptic:

Classification is mechanical first. Path rules and diff facts set the tier: migrations/ is red, anything under auth or billing is red, a dependency manifest change escalates, a negative coverage delta escalates. An agent assigning its own color is grading its own homework, and misclassification is precisely the blind-spot error models are worst at. The agent may propose. The rules dominate.

The ratchet. Anything (rule, agent, human) can escalate a tier. Only a human with maintain rights can lower one, and lowering requires a written reason. The label is itself a one-way door. In code it's a single decision: apply max(existing, computed), never the fresh computation on its own, or a routine push after a human escalation silently un-escalates the PR.

Green-washing is the attack. A red change wearing a green diff. The copy tweak on a form that feeds billing. The dependency bump that changes auth behavior. Non-overridable escalators exist for that case specifically, which makes the escalator list the part of the config worth arguing over.

The matrix guards its own config. The tier rules file and the workflow enforcing them both sit under red paths, so changing the gate requires human review. A gate you can edit through the green lane isn't a gate.

It's calibrated, not decreed. Every bug traced back to a merged PR gets tagged with that PR's tier at merge time. Green leaking means a boundary was wrong, so that area gets promoted with the incident attached as evidence. Humans own where the red line sits. Agents produce the evidence about whether it sits in the right place.

What this looks like when it's already running

I run a smaller version of this design today, pointed at a different risk. On the machines where my agents work, package installs pass through a gate that refuses any version published within the last fourteen days, any package under a year old, anything from an unfamiliar forge, and anything under ten thousand weekly downloads. It fails closed. The agents can't grant themselves an exception. The allowlist is mine. The last time it fired, it stopped an unattended build from fetching and running a Cloudflare CLI release that had been on npm for zero days. The build fell back to the repo's pinned copy and carried on.

Nothing about that gate is clever. The same three properties carry it as the matrix does: a mechanical floor the agent can't argue its way past, escalation that only a human reverses, and a rule that applies to the tooling itself. Supply chain was the easy case to justify, because that failure mode has a ten-figure damage bill and a Wikipedia page. Review is the same design pointed at the thing that actually consumes my week.

The ask: one repo, one month

None of the above deserves belief because it's well argued. Here's the falsifiable version, four weeks, one repo.

Week 0. Write the tier rules from that team's own incident history, not from my examples. Mirror red paths into CODEOWNERS so red leans on native branch protection instead of on custom code that can break silently.

Week 1. Labels only, nothing enforced. Boundary disagreements get filed against the rules file while disagreeing is still free. If more than half of PRs land red, the boundaries are wrong and the system is routing attention nowhere, which is the status quo with extra steps.

Week 2. Agent review on every tier, one invocation per lens with fresh context, structured output. Publish token cost per PR next to the baseline human review minutes, in public, both directions.

Week 3. Enforcement. Red requires human approval. Yellow requires agent passes plus a human comment. Green auto-merges on clean checks, the one tier where reclaimed minutes are unambiguous, because the human minutes go to zero. Then seed three test PRs: one touching migrations, one benign docs change, one modifying the gate workflow itself. They should come out red, green, red. If the third one comes out green, stop and fix the escalators before continuing.

Week 4. The numbers. Escaped defects per tier against baseline, review minutes per PR against baseline, token cost per PR, and every boundary move with the incident that triggered it.

Baseline captured before any of it starts, or the pilot can't be falsified, and an unfalsifiable pilot is a marketing campaign.

Kill criteria, written before the data exists, because criteria written afterward are decoration. Kill it if green-tier escaped defects exceed the pre-pilot defect rate, or if the team is still disputing more than a fifth of labels in week four, which would mean the mechanical floor doesn't fit that codebase. On an ambiguous result, extend a month rather than expanding to more repos. Expanding on ambiguity converts the proposal into exactly the unmeasured judgment it was arguing against.

What isn't proven yet

• No measured catch rate for agents reviewing agent-authored code. The pilot's per-tier escape counts are the beginning of that number, not the number. I won't quote a defect-reduction multiple until there's at least a quarter of data behind it.
• No generic human error rate borrowed from industrial reliability studies. Those describe procedural tasks under different conditions. The software numbers carry the argument alone.
• Agent-proposed tiers are out of scope for the pilot. Adding a judged input before the mechanical one has earned trust is one variable too many.

The method, demonstrated on itself

An earlier draft of this argument went to an adversarial critic agent for about a dollar. Twelve objections came back. One was a contradiction between two sections that I'd have published without noticing. Four numbers came out because they couldn't survive a hostile reader, including one I was attached to.

The draft was mine. The skeptical pass was nearly free. The expensive part, deciding which of the twelve objections were right, took twenty minutes of the priciest tokens I've got. Exactly where they belonged.

Sources

Every stat above, with somewhere to check it. Where a figure lives in a book, the link goes to a page that reproduces it, with the book named underneath.

• Delivered code carries 15 to 50 defects per thousand lines. Reproduced with attribution by Sogeti Labs. Original: Steve McConnell, Code Complete, 2nd ed., chapter 20.
• Informal review catches 20 to 35 percent of defects. Formal inspection reaches 55 to 60. The inspection figures, quoted with attribution, are at Coding Horror. The full table, including the informal-review row, is McConnell, Code Complete, 2nd ed., Table 20-2 ("Defect-Detection Rates"), compiled from Capers Jones and others. Jones's own defect-removal data puts inspections higher, 65 to 85 percent, so the numbers used here are the conservative ones.
• Reviewer effectiveness drops past about 60 minutes and 400 lines. Jason Cohen et al., Best Kept Secrets of Peer Code Review (SmartBear), from the Cisco Systems study of roughly 2,500 reviews across 3.2 million lines.
• Model-side catch rates were measured on deliberately inserted bugs. Nat McAleese et al., "LLM Critics Help Catch LLM Bugs" (OpenAI, 2024).