Fifty percent fewer errors is a ceiling, not a floor

There is a figure doing the rounds in every slide deck about specification-driven delivery: controlled studies have found that human-refined specifications cut errors by up to 50 percent. It is a real finding and it points at something true.

It is also being read backwards. My position: that number is a ceiling reached under favourable conditions, not a return you can plan against, and the more damaging mistake is assuming the remaining errors are the same kind that more specification effort would remove. They are not. That misreading is why spec programmes get enthusiastic funding in month one and get quietly cancelled in month four.

Read the sentence precisely

Three words are carrying the weight.

“Up to” is a ceiling. It describes the best result observed, not the median one, and the gap between best and median in this kind of study is usually large.

“Controlled” means the conditions were arranged. Someone who knew what they were doing wrote the specifications, on tasks selected to be specifiable, with the refinement effort actually spent rather than squeezed into the end of a planning meeting. Your Tuesday afternoon is not that.

“Errors” is a category defined by the study, not by your incident register. It typically means the output failed to do what was asked. That is a real class of failure and it is not the class that wakes you up at three in the morning.

None of this makes the finding worthless. It makes it a direction rather than a forecast, and direction is what you should use it for.

How the figure gets used

• "Specs halve our defects"
• Business case built on a 50 percent reduction
• Success measured against that number in a quarter
• More effort assumed to give more return
• Programme judged a failure at 15 percent

What it supports

• "Precision in specs removes a specific class of rework"
• Business case built on the rework you can already attribute
• Success measured as a change in the mix of failure types
• Return assumed to flatten quickly
• Programme judged on whether the targeted class shrank

The half you cannot specify away

Here is the substantive claim, and it is the one worth arguing about. The errors that specification removes and the errors that remain are not two portions of the same pile. They have different causes.

Specification removes ambiguity errors: the builder did something reasonable that was not what you meant, because what you meant was never written down. Sharpen the criteria and that class genuinely shrinks.

What is left is dominated by something else. Context blindness, meaning output that is entirely internally coherent and incompatible with the repository it lands in: references to APIs that do not exist, violations of local conventions, assumptions about a runtime that is not the one you run. A perfect specification does not touch this, because the specification describes the destination and the failure is about the terrain.

AmbiguityThe request was underdetermined. Specification is the correct fix and the returns are immediate.

Context mismatchThe request was clear, the environment was not known. Fixed by what the builder can see, not by what the ticket says.

IntegrationEach change is correct alone and wrong together. Only visible at the point where they meet, which is after both are written.

Wrong intentThe specification was precise, checkable, agreed, and described the wrong thing. No process fixes this; only contact with a user does.

Why more effort stops paying so fast

The returns flatten because ambiguity is finite. A ticket has a limited number of genuinely load-bearing unknowns, and once you have resolved them, further specification produces length rather than clarity.

You can see the point of diminishing returns in a refinement session if you watch for it. Early in the discussion, questions change what will be built: what happens to accounts already mid-export, whether internal tokens are exempt, what the failure mode is when the downstream is down. Later, questions stop changing anything. They produce more words that everybody already agreed on.

That second half of the meeting is where spec programmes acquire their reputation as bureaucracy, and the reputation is deserved. The move is not to spec harder. It is to stop when the questions stop changing the answer, and to move the remaining budget to verification, which is where the other three rows in that figure get caught.

What to promise instead

If you are making the case internally, promise something you can actually be judged on.

Attribute your current rework to those four causes for a month, honestly, even roughly. Then promise a reduction in the first row only, and say out loud that the other three rows are not what this programme is for. You will be promising a smaller number, on evidence from your own organisation rather than from a controlled study of somebody else’s, and you will still be there in month six.

There is a bonus you can promise with more confidence than the error reduction: specifications that are precise enough to check make failures visible earlier. Even where they do not prevent the error, they change discovery from production to review, and that shift is often worth more than the reduction anybody was arguing about.

Where this breaks down

Ceilings do get reached. I have argued that “up to 50” describes favourable conditions, and favourable conditions are constructible. A disciplined team on a well-understood domain with a strong definition of done may genuinely sit near the top of that range, and telling them to expect less would be wrong.

My four-row taxonomy is not a measured thing. It is a way of thinking that I find holds up in incident reviews, not a validated classification, and the boundaries between rows are fuzzy in practice. A great deal of what I would call context mismatch could be argued as ambiguity by someone who thinks the environment should have been specified.

Specification may reach context blindness after all. If criteria routinely name the local conventions, the existing interfaces, the module boundaries to respect, then the specification is carrying context and my clean separation dissolves. That is roughly what context engineering claims, and it is a serious argument against my position.

Discouraging a big promise has a cost. Programmes get funded on ambition. Advising people to promise a smaller number from their own data is safer and may mean the work never gets funded at all, in which case my caution has produced a worse outcome than the overclaim would have.

Attribution of rework is genuinely hard. I have recommended attributing a month of rework to four causes as if that is a light exercise. It requires judgement calls on every item, made by people with an interest in the answer. The data you get will be soft, and I would not build a business case on it without saying so.

The takeaway

The 50 percent figure describes what precision can do to one class of error under good conditions. It is not a baseline, it does not scale with effort, and it says nothing about the failure modes that dominate once ambiguity has been cleared.

Spend specification effort until the questions stop changing the answer. Then stop, and move the rest of the budget to verification, because that is where the other half lives.

If you take one thing into next week: take the last ten pieces of rework your team absorbed and sort them into ambiguity, context mismatch, integration, and wrong intent. The shape of that pile tells you how much of the 50 percent was ever available to you.