Code review: from gatekeeper to sampler
Reading every diff was always a fiction, and agents turned it into an obvious one. What replaces it is not less rigour but rigour aimed at classes of change rather than individual ones.
Open the pull request queue on a Friday afternoon at a team that has been running agents for a few months and count what is in it. Not the ones assigned to you. All of them. If the number is under ten you are probably fine. If it is forty, look at how long the average one has been sitting, and then look at the approvals from this week and ask yourself honestly how many of them involved someone reading the whole diff.
I know what the answer is, and so do you, because I have asked a lot of people this question and the answer is always some version of the same sheepish shrug. The approvals are happening. The reading is not. Not entirely. Not the way it was in 2019, when a 300-line diff landed twice a day and you actually went through it.
The interesting thing is that the reading was never as thorough as the ritual implied, even then. There is a body of research on this stretching back years, and the consistent finding is that reviewer effectiveness collapses past a few hundred lines, that most defects found in review are found in the first twenty minutes, and that a large share of approvals on large diffs are effectively rubber stamps. Everyone in the industry has known this and everyone has kept the ritual, because the ritual was cheap enough to sustain and the alternative was unclear.
Agents did not create the problem. They removed the ability to ignore it. When the volume goes up by a factor of five or ten, the gap between “we review every change” and what actually happens gets too wide to keep a straight face about.
Gatekeeping was a throughput bet
Consider what the gatekeeper model assumes. One human reads one change and decides whether it may pass. This works when the ratio of change volume to reviewer attention is favourable, and it degrades gracefully when it is slightly unfavourable (things sit in the queue a bit longer, everyone grumbles) and catastrophically when it is very unfavourable.
Catastrophically, specifically, in this way: the reviewer does not stop approving. They stop reading. The gate stays visibly in place while ceasing to function, which is worse than removing it, because everyone downstream still behaves as though changes have been reviewed. A gate that is known to be open is a risk you can manage. A gate that appears closed and is not is a risk you cannot even see.
The reviewer does not stop approving. They stop reading.
The gate stays visibly in place while ceasing to function, which is worse than removing it, because the organisation is still making decisions on the assumption that it works.
So the first move is to stop treating “review everything” as the standard and admit that what you actually have is a sampling process with an unexamined sampling strategy. Because that is exactly what it is. Your reviewers are already sampling. They are sampling by whim, by fatigue, by which pull requests happen to be small, by who asked nicely in Slack. That is the worst possible sampling strategy: it is biased, undocumented, and unmeasurable.
Once you accept that you are sampling, you can sample deliberately. And deliberate sampling is a genuinely strong technology. It is how manufacturing quality control works, it is how auditing works, it is how epidemiology works. None of those fields inspects every unit and none of them considers this a failure.
Sampling by accident
- Whatever fits in the time available
- Selected by queue position and familiarity
- Uncorrelated with risk
- Nobody admits it is happening
Sampling deliberately
- A strategy chosen in advance
- Weighted by consequence
- How manufacturing QC and auditing work
- Defensible to anyone who asks
Risk-weighted, not uniform
The core of a sampler model is that not all changes deserve equal attention, and the differences are large enough to be worth encoding.
Some changes are high-consequence in a way that is knowable in advance and has nothing to do with how good the code is. A change touching authentication, payments, permission checks, data deletion, migration scripts, cryptographic material, external API contracts, or anything that runs with elevated privileges is in a different class from a change that adds a column to an internal admin table. Not because agents are worse at auth (they may be better at it than your average engineer, honestly) but because the cost distribution of being wrong is completely different.
That suggests something like three tiers, though the exact shape varies:
Always reviewed by a human, carefully, no exceptions. Security boundaries, money, data destruction, anything irreversible, anything that changes a contract other people depend on. This tier should be small, ideally under fifteen percent of changes, and it should be defined by path and pattern rather than by judgement, so it cannot be quietly skipped on a busy Friday.
Gated by automation, sampled by humans. The large middle. Every change must pass its acceptance criteria, its tests, its static analysis and its policy checks. Humans review a random sample, plus anything the automation flags as unusual. The sample rate is a dial you set based on how much you trust the class, and importantly the sample is random, because a predictable sample is a sample you can route around.
Automated only. Genuinely low-consequence, highly-patterned changes: dependency bumps that pass the suite, string updates, generated client regeneration, test-only additions. Reviewed by policy, not by a person.
The thing that makes this work is not the tiering. It is that the tiering is written down, applied mechanically, and measurable. You can look at a quarter and say “we sampled twenty percent of tier two and found defects in three percent of the sampled changes,” and that is a real statement about your quality posture. Compare it to “we review every pull request,” which is a statement about your intentions.
What a human reviewer is actually good at
If automation handles correctness against stated criteria, what is left for the person? This is the question that determines whether the sampler model feels like a demotion or a promotion, and I think the honest answer is that humans get the more interesting half.
Whether the criteria were right. Automation checks the change against the acceptance criteria. Nothing checks whether the acceptance criteria described the thing anyone wanted. That judgement is the highest-value review activity in an agentic team and it is nearly absent from most review checklists, which are all oriented toward implementation correctness.
Architectural drift. Any individual change can be locally reasonable and the aggregate can be a mess. Agents are particularly good at producing locally reasonable changes, and particularly bad at noticing that this is the fourth slightly different retry implementation in the codebase. This is a review question that operates over a set of changes rather than one diff, which means it does not fit the pull-request-shaped review form at all, and which is a strong argument for a periodic architectural review that is separate from per-change review.
The unstated constraint. The thing that was obvious to everyone and therefore never written down. It will keep not being written down, because that is what obvious means, and a human is currently the only mechanism for catching it.
Blast radius reasoning. Not “is this correct” but “if this is wrong, what happens, and how would we know.” Agents will happily produce a change that is correct and undetectably fragile. Humans are, for now, better at the counterfactual.
Read that list and notice: every item is a judgement, none is a correctness check, and all four scale better than line-by-line reading because they operate on structure rather than on volume. A reviewer doing those four things on a sampled subset is contributing more than a reviewer skimming everything.
The problem of provenance
There is a practical prerequisite that teams tend to hit hard, and it is unglamorous: to sample intelligently, you have to know what you are sampling from.
That requires knowing, for every change: which work item it belongs to, what the acceptance criteria were, whether they were met, who or what authored it, what tier it falls in, and whether it was sampled. A surprising number of teams cannot answer the first of those questions reliably, because the link between a pull request and a ticket is a branch naming convention that half the agents do not follow.
Without that link, every downstream thing is guesswork. You cannot report review coverage by risk tier if you cannot classify changes. You cannot verify that a change met its criteria if you cannot find its criteria. This is the specific gap that a control layer like GroundTruth exists to close: linking every pull request to its work item, checking the change against machine-verifiable criteria before “done” is allowed, and keeping a signed record of what was checked and by what. Whether you buy that or build it, you need it, because the sampler model without provenance is just the gatekeeper model with the pretence removed.
The cultural part is harder than the technical part
Everything above is straightforward to implement and difficult to adopt, and the difficulty is not really about risk.
Code review is a social institution. It is how teams transmit standards. It is where a senior engineer teaches a junior one what good looks like, in small corrections, over months. It is a large part of how people build a mental model of a codebase they did not write. And it is one of the few remaining places where engineers read each other’s work at all.
Cut the volume of human review by eighty percent and you do not just cut defect detection by some amount. You cut the apprenticeship channel. New engineers stop absorbing the codebase through reading it. Standards stop propagating by osmosis and have to propagate by documentation, which they do much worse. The team’s shared mental model thins out, and nobody notices for two quarters, and then a decision gets made badly because three people had three different pictures of how the system works.
Teams that do this well seem to replace the channel rather than mourn it: deliberate reading sessions where a group reads a significant change together, rotation so that people see parts of the system they do not own, written architectural decision records because the oral tradition is gone. Teams that do it badly just have less review and pretend that was the only thing being lost.
There is also a status question. Approving pull requests was a marker of seniority and trust. Telling a senior engineer that most changes will now be approved by policy is heard, correctly, as a change in their role. Being explicit that the role moved up rather than away (criteria quality, architecture, risk tiering) matters more than any tooling decision you will make.
Where this breaks down
The honest limits, several of which are substantial.
Sampling assumes independent defects and agent output is correlated. This is the strongest objection and it deserves more attention than it usually gets. Statistical sampling works because a defect in unit 400 tells you little about unit 401. Agent output does not behave that way. If a model has a systematic misunderstanding, or your prompt scaffolding encodes a bad pattern, that error appears in every change of that shape, and a random sample will find it exactly as often as it appears, which sounds fine until you realise you needed to find it the first time. Correlated defects are a different statistical problem than the sampling literature assumes. The mitigation is to sample by class rather than uniformly, and to treat any defect found as evidence of a class-wide problem rather than a one-off, but that is a mitigation, not a solution.
Risk tiering is only as good as your file paths. The tiering above assumes you can identify high-consequence changes mechanically. In a well-structured codebase you can. In one where the payment logic is spread across nine modules and a utility file called helpers.py, you cannot, and your tier one will silently miss the changes it exists to catch. Fixing this means fixing the codebase structure, which is a much larger project than fixing review.
Automated gates have their own failure modes and they fail silently. A test suite that passes tells you the assertions passed. If the assertions were generated alongside the code, that is a weaker signal than it looks. Leaning harder on automation means leaning on something whose trustworthiness is exactly the open question.
Some domains cannot sample. Avionics, medical devices, anything where a regulator requires documented review of every change: the tier one bucket is one hundred percent by law and no amount of sampling theory helps. If that is you, the honest answer is that your throughput ceiling is your review capacity, and the correct response is to use agents to make each review faster (better change descriptions, better test evidence, smaller diffs) rather than to review less.
And smaller diffs are not free. A lot of the advice here assumes changes are small and well-scoped enough to classify and review quickly. Agents can produce enormous diffs very easily, and “keep changes small” is much harder to enforce on a system with no fatigue and no sense of proportion. Many teams find the diff size problem is the actual bottleneck, not the review model.
The takeaway
Review as a universal gate was already a partial fiction, and agent volume made the fiction untenable. The choice is not between rigour and speed. It is between undocumented sampling by fatigue and deliberate sampling by risk.
The move is to tier changes by consequence rather than treat them uniformly, automate the correctness checks, sample the middle randomly, and redirect human attention to the four things humans are actually better at: whether the criteria were right, whether the architecture is drifting, what was obvious and unstated, and what happens when this is wrong.
Which lands us on an uncomfortable dependency. Most of this rests on automated checks, and the most important automated check is the test suite. So what exactly is a passing test suite worth when the same system wrote both the code and the tests? That is the next piece.