A spec a machine cannot check is a wish
The test for an acceptance criterion is not whether a person agrees with it. It is whether anything could ever disagree with it. Most criteria fail that test, and the failure is invisible until the work comes back wrong.
Here is a criterion from a real refinement session, or near enough to one that you will recognise it: “the export must complete in a reasonable time for large accounts.”
Everyone in the room nodded. It went into the ticket. It is not a criterion. It is a wish with a tick box next to it.
My position: the only useful test of an acceptance criterion is whether a machine, handed the criterion and the change, could return pass or fail with a reason. Not whether it does today, and not whether you have automation for it. Whether it could in principle. Criteria that fail that test are not weaker criteria; they are a different kind of object entirely, and treating them as the same thing is where most agentic rework starts.
Falsifiability is the whole test
A criterion is a claim about the finished system that could turn out to be false. If nothing could ever count as failing it, it carries no information.
Run three examples through that filter.
“The API should be secure.” Nothing counts as failing this until an incident does. It is a value, not a criterion.
“Rate limits are enforced per client, not globally.” A machine can locate the enforcement point, read the key, and tell you which one it is. Falsifiable.
“The UI should feel responsive.” Nothing counts as failing it. But it can be rewritten into something that does: interaction returns first paint under a stated budget on a stated device class. Now it is falsifiable, and you have also had the argument about what “responsive” meant, which is the argument you were avoiding.
That last example is the important one. Most wishes are not empty. They are compressed. Somebody in the room knows what they mean and has not written it down, because among humans compression is efficient. Everyone decompresses from shared context and gets close enough.
Wish
- Handles errors gracefully
- Works for large accounts
- Backwards compatible
- Follows our conventions
- Does not break anything else
Criterion
- Upstream timeout returns 503 with a Retry-After header, no partial write
- Accounts above the stated row count complete within the stated budget
- Existing v1 request shapes return unchanged response shapes
- No new direct database access outside the repository layer
- Named downstream consumers listed, each with its contract check
Why the wish survives refinement
Wishes pass review because they are unobjectionable. Nobody argues against graceful error handling. The criterion that says “returns 503, no partial write” invites an argument about whether 503 is right, and arguments take time in a meeting where six other tickets are waiting.
So the wish gets waved through, and the disagreement is not resolved. It is deferred, and it lands on whoever is closest to the deadline. That used to be a senior engineer with context, who quietly resolved it well and never mentioned it. It is now a builder with no judgement and no memory of the last outage, resolving it plausibly and confidently in whatever direction the surrounding code suggests.
This is the mechanism behind a lot of what gets called context blindness. The output is internally coherent, matches the ticket as written, and is incompatible with what the organisation actually needed, because the ticket as written was never the specification. It was a pointer into a body of understanding, and the pointer got followed by something that does not hold the body.
Checkable is not the same as automated
The objection I hear most is that this demands test automation for everything, which nobody can afford. It does not.
The requirement is that the criterion has a truth value, and that the evidence for it is reachable from the change. Several kinds of check satisfy that.
Sorting criteria into those four buckets during refinement takes a couple of minutes per ticket and does something useful on its own: it makes the unsortable ones obvious immediately, and the unsortable ones are the wishes.
The number people quote, and what it actually says
Controlled studies have found that human-refined specifications cut errors by up to 50 percent. That figure is doing a lot of work in a lot of slide decks at the moment, usually as an argument for spending more time in refinement.
Read it precisely. “Up to” is a ceiling under favourable conditions, not a return you should plan against. And the mechanism it points to is not effort, it is precision. A refinement session that spends an extra thirty minutes producing more prose has added length, not truth value. The gain comes from resolving the specific ambiguities that would otherwise be resolved badly downstream, which is a small, targeted activity, not a longer meeting.
Where this breaks down
Exploratory work genuinely resists this. When you do not yet know what good looks like, demanding falsifiable criteria up front produces confident nonsense: precise targets invented to satisfy a process. The honest move on a spike is to write down the question you are trying to answer and skip the criteria entirely, rather than dressing a wish up as a measurement.
Falsifiable criteria can be gamed. State a latency budget and you may get a cache in front of a broken query. Every precise target creates an incentive to satisfy the target rather than the intent, and precision makes that easier, not harder. The defence is criteria that constrain the mechanism as well as the outcome, and that costs more to write than I have made it sound.
Some of the best criteria I have seen were unfalsifiable. “A new engineer should be able to find this without asking.” No machine will ever evaluate that, and it has prevented more bad designs than any latency budget. There is a category of intent that survives only in prose, and a rule that deletes it is a rule that costs you something real.
The rewrite has to happen somewhere, and somebody has to pay for it. I have described turning wishes into criteria as though it is free. It is a genuine transfer of work into refinement, done by the people with the least slack in the organisation. If you add it without removing something, it will quietly stop happening by the third sprint.
Volume may beat precision. If generation is cheap enough, there is a real argument for shipping against loose criteria, seeing what comes back, and correcting. For low-risk surfaces with fast feedback, that is probably right, and my position applies with much less force there than I would like it to.
The takeaway
An acceptance criterion earns its place by being capable of being wrong. If nothing could count as failing it, it has not specified anything, and the decision it was meant to settle has been quietly forwarded to whoever is furthest from the context.
You do not need automation for all of it. You need every criterion to have a truth value and a reachable source of evidence, which is a writing problem before it is a tooling problem.
If you take one thing into next week: open your last three closed tickets and mark every criterion that nothing could have failed. That count, per ticket, is the best predictor you have of where the next round of rework comes from.