Change failure rate is the one to watch

Suppose a team had four production incidents last quarter out of two hundred deploys. That is a two percent change failure rate, which is a good number by any banding you care to use. This quarter they have five incidents out of six hundred deploys. Under one percent. The chart shows a healthy downward trend and somebody puts a green arrow on it.

Five is more than four. The number of times this quarter that something went wrong badly enough that a human had to intervene went up by a quarter. The rate went down by more than half. Both statements describe the same three months.

I have come to think change failure rate is the most important of the four keys under agentic development, and simultaneously the one being read most carelessly. Those two facts are related. It matters because it is the only one of the four that points at whether the extra speed is costing you anything, and it is read carelessly because it is a ratio and its denominator just multiplied.

Why this one survives the transition

Go back to what happened to the other three.

Deployment frequency rose because you can now produce more changes per decision, which tells you about production volume rather than delivery health. Lead time fell because the interval we measure collapsed while the interval we do not measure did not. Time to restore, as I will argue in the next piece, holds roughly flat while the work behind it gets harder.

Change failure rate is different in kind, because its numerator is anchored to something agents cannot inflate. A production incident is a fact about the world. A revert is a fact about your repository. Neither of those events is produced by generating more code. If agentic development is genuinely degrading your quality, that degradation must eventually show up as more of these events, and no amount of merge volume creates them out of nothing.

That anchoring is rare and valuable. Most of what we can measure under the ADLC is either a count of artefacts, which inflates, or a self-reported state, which drifts. Failures are neither. They are unambiguous, they are expensive, and somebody always notices.

The problem is entirely in the denominator.

The denominator problem, stated plainly

Change failure rate is failures divided by changes. Agentic development multiplies changes without any corresponding multiplication of failures, because most of the additional changes are small, mechanical or low risk.

So the rate falls. It falls even if your absolute failure count is rising. It falls faster the more enthusiastically you adopt agents. And it falls in a way that looks exactly like improvement, because for the previous twenty years a falling change failure rate genuinely was improvement, since the denominator moved slowly and any movement in the ratio came from the numerator.

There is a second, nastier version of this. The additional changes are not a random sample of your changes. They are systematically the safe ones: dependency bumps, test coverage, small refactors, the kind of tidying that used to sit undone because it was not worth a human afternoon. You have diluted your change population with low-risk changes. The rate is now dominated by them. Your risky changes are as risky as they ever were, and they are a smaller fraction of the total, so their contribution to the ratio is smaller.

The rate is measuring dilution, not quality.

Last quarter    4 incidents / 200 deploys  =  2.0%
This quarter    5 incidents / 600 deploys  =  0.8%
Rate:      down by more than half
Incidents: up by a quarter

Read the numerator

The fix is unglamorous: stop leading with the rate and lead with the count.

Absolute failures per week is a worse-looking metric. It does not normalise, it is not comparable across teams of different sizes, and it will not fit into the elite-high-medium-low banding that makes DORA presentable to executives. It is also the number that tells you whether things are getting worse, and I would trade all four of those properties for that one.

In practice I would look at three things together.

Absolute incident count per unit time, on a chart with a long enough window to see through the noise. This is your ground floor. If it is rising, something is wrong regardless of what the rate says.

Revert count and revert rate, tracked separately from incidents. A revert is a cheaper, earlier signal than an incident. It means something got merged that should not have been, and it got caught before or shortly after reaching users. Under agentic development I would expect reverts to rise faster than incidents, because more changes flow and the review layer is under more pressure. That is not necessarily bad news. Reverts are the system working. But a sharply rising revert count is a leading indicator of an incident count that has not risen yet.

Failure count restricted to the risky population. If you can classify changes at all, even crudely by which paths they touch, compute the failure rate over the subset that touches things that matter. That subset did not multiply the way the total did, so its rate is still meaningful. This is the closest thing to a like-for-like comparison with your pre-agent numbers.

The most defensible basis for this metric is reverts over merges, stated on the metric’s face. Reverts are visible in every source control provider, whereas incidents live in whatever incident tool a given organisation happens to run and are not consistently available. That choice makes the metric computable everywhere, and it makes it an early-warning proxy rather than a measure of user-visible harm. It is a weaker claim than the DORA definition. It is also the claim the data supports, and saying so on the surface is the only way the reader can apply the right discount.

The thing that actually changes under the ADLC

Here is what I think is genuinely different about failure under agentic development, beyond the arithmetic.

Failures used to correlate with complexity. The changes that broke things were the hard ones: the tricky migration, the concurrency fix, the refactor that touched forty files. Review effort was allocated accordingly, roughly by instinct, and it worked because the difficulty of writing a change and the risk of that change were correlated. Hard to write meant hard to get right meant worth reviewing carefully.

That correlation is weaker now. An agent produces a subtly wrong forty-file refactor as fluently as a correct one, and both arrive looking equally finished: consistent style, tests present, description well written, no smell of struggle. The signals reviewers used to triage by, hesitancy in the commit history, a messy diff, a long PR description apologising for something, are gone. Everything arrives looking like the easy case.

So the risk is no longer where the effort was. It is distributed differently, and it is harder to see. That is a claim about review, not about metrics, but it shows up in metrics as a failure population that is less predictable than it used to be. If your incidents used to cluster around a few known dangerous areas and now arrive from everywhere, that is worth noticing, and no ratio will show it to you. You have to look at where the failures actually came from.

What good looks like

If I were setting a target under the ADLC, I would not set one on the rate at all. I would set two commitments.

The first is that absolute failure count must not rise as throughput rises. This is a hard commitment and a fair one. If you have doubled your change volume and your incident count is flat, you have genuinely improved, and the rate will show it too. If your incident count has risen with volume, you have bought speed with reliability, which may be a trade you want to make for a quarter but should never be a trade you make by accident.

The second is that every failure gets attributed to a change, and the attribution is recorded. Not for blame. For the population analysis above. If you cannot say which changes caused your failures, you cannot tell whether risk has redistributed, and you are left with a ratio that is being pushed around by dilution.

Neither of these needs new tooling. Both need somebody to care about the numerator.

Where this breaks down

Several places, and one of them is serious.

The serious one is that absolute counts do not normalise, and I have just recommended managing by them. If a team grows, or takes on a second product, or inherits a legacy service, its absolute incident count will rise for reasons that have nothing to do with agentic development or quality. Manage by absolute count without adjusting for scope and you will punish teams for taking on work. The rate exists for a reason, and the reason is real. My argument is that under a rapidly inflating denominator the rate temporarily stops working, not that normalisation is a bad idea in general. If your change volume stabilises at a new plateau, go back to the rate.

The second limit is that reverts are a noisy proxy for failure. Some reverts are failures caught early. Some are a change being rolled back because priorities shifted, or because a flag was flipped, or because someone reverted to un-break a broken main and then re-landed the same change twenty minutes later. A revert-derived change failure rate counts all of these identically. It will overstate your failure rate in absolute terms and, more importantly, it will move for reasons unrelated to quality. Treat it as a direction, not a level.

Third, the claim that agents produce risk that is invisible to reviewers is the part of this piece I hold least firmly. It matches what I see and what people describe, but the alternative explanation is that reviewers are simply under more load and would catch these things given time, which is a capacity problem rather than a legibility problem. Those two have different fixes. Capacity problems are solved by reducing volume or adding reviewers. Legibility problems are solved by changing what review looks at. I would not spend heavily on the second until I had ruled out the first.

Fourth, and most awkwardly: a team that responds to this piece by heavily instrumenting failure attribution may find that the act of attribution changes behaviour in bad ways. Engineers who know every incident gets traced to a change will make smaller, safer, less ambitious changes, and some of the resulting caution is pure loss. The best version of this is a blameless population analysis nobody’s performance review touches. The worst version is a defect-tracking regime with extra steps, and the distance between them is entirely about how the organisation behaves, not about what the tool computes.

The takeaway

Change failure rate is the one of the four keys that agentic development cannot flatter through the numerator, which makes it the most valuable signal you have. It can absolutely be flattered through the denominator, and under the ADLC it will be.

So read the count, not the rate. Track reverts separately as an early warning. Restrict the rate to the risky subset of changes if you can classify them at all. And commit to absolute failure count holding flat as throughput rises, because that is the only version of this metric that speed cannot fake.

The next piece is about time to restore service, which is the most robust of the four keys and the one where the hidden work, diagnosis, has quietly got much harder in an agent-heavy codebase.