Sequencing, criteria before automation, always

Here is a failure I have now seen described enough times to consider it a genre.

A team wires up automated status transitions. It works. Tickets close themselves on merge, the board is finally current, everyone is pleased for about three weeks. Then somebody notices that the number of tickets closing per sprint has gone up by a third, while the number of complaints from the support queue has also gone up, and the two facts turn out to be the same fact. Tickets were closing on merge. Merging was not the same as done. Nobody had said what done meant with enough precision for the difference to be visible, so the automation faithfully encoded the ambiguity and ran it at speed.

The board was now accurate about a thing that did not mean what everyone assumed it meant.

That is the argument for this article in one paragraph. Automation does not resolve ambiguity. It preserves ambiguity, makes it fast, and wraps it in an audit trail that makes it look like a decision somebody made on purpose.

Why the temptation is so strong

The reason everyone inverts this order is not stupidity. It is that the two pieces of work have wildly different textures.

Automation is a Tuesday. You connect two systems, you configure some rules, you watch it work, and there is a visible artefact at the end that you can show to the person who funded it. It feels like engineering because it is engineering.

Criteria work is six weeks of arguments in refinement about what “done” means for a feature that four people understood differently. There is no artefact. Nobody demos it. It generates conflict, because it forces disagreements into the open that were previously resolved by everyone quietly assuming they agreed. It is unpaid, unglamorous and politically expensive.

So teams do the Tuesday. Every time. And the criteria work moves to a backlog called “process improvements” where it dies.

Automation work

• A Tuesday
• Visible artefact at the end
• Demoable to whoever funded it
• Feels like progress
• Does not resolve any ambiguity

Criteria work

• Six weeks of refinement arguments
• No artefact
• Nobody demos it
• Generates conflict
• Is the actual transition

The uncomfortable truth is that the criteria work is the actual transition and the automation is the delivery mechanism. If you only have capacity for one of them, do the criteria work and update your tickets by hand. You will be better off than the reverse.

What “criteria first” concretely means

This is where the advice usually goes abstract, so let me be specific about the bar.

A definition of done is sufficient when a competent person who was not in the planning conversation could read it and decide, unambiguously, whether a given change satisfies it. Not “would probably agree.” Decide.

That bar rules out most of what currently sits in most backlogs. It rules out “the checkout flow should work correctly.” It rules out “improve performance of the search page.” It rules out “handle errors gracefully.” Each of those is a sentence that a human developer would have completed from context and taste, which is precisely the thing you can no longer rely on.

What passes the bar tends to have four components, whether or not you write it in a formal structure:

• A subject. Which thing is changing, specifically enough to locate it.
• An observable outcome. Something you could check, not something you could feel. “Returns a 409” rather than “handles the conflict properly.”
• A boundary. What is explicitly out of scope. This is the component teams skip and it is the one that prevents an agent from cheerfully rewriting four adjacent modules.
• A verification path. How you would know. A test, a query, a manual step, an observable log line. If nobody can say how you would check it, it is not a criterion, it is a wish.

You do not need Gherkin. You do not need a template. Given-when-then helps some teams and turns into ritual for others. The four components are the thing; the format is taste.

The order, concretely

If you take one thing from this article, take this sequence.

One: score before you rewrite. Find out what proportion of your active and near-term tickets pass the bar. Do this before you improve anything, both because it is your baseline and because the number itself is usually persuasive enough to buy you the time to do the work. Teams guess they are at seventy percent. They are rarely above forty.

Two: rewrite only the intersection of bad and imminent. Criteria on work more than six weeks out will change before anyone acts on them. Spending refinement capital there is waste.

Three: build the habit in refinement, not in a project. The moment criteria improvement becomes a work stream with a deadline, you get criteria written to satisfy the work stream. What you want instead is a permanent change to how a ticket leaves refinement, which is a habit, and habits are installed by making the check cheap and constant rather than large and occasional.

Four: only now, turn on reconciliation. Read ground truth, update the record, keep everything additive and reversible.

Five: only after reconciliation is boring, turn on anything that gates. Merge checks, done-criteria enforcement, anything that can block a person. Gating on a definition of done you have not verified is how you teach a team to route around the gate within a fortnight, and once they have learned that, you will not get it back.

ScoreWhat proportion of active tickets pass the bar. Your baseline, and usually a shock.

Rewrite the intersectionOnly what is both bad and imminent. Criteria six weeks out will change before anyone acts on them.

Build the habit in refinementAt the exit, not as a project. A project produces criteria written to satisfy the project.

Turn on reconciliationAdditive, reversible, no gating. This step does not depend on criteria quality.

Only then, gateMerge checks and enforcement. Gating on criteria you have not fixed teaches people to bypass gates.

The one legitimate exception

There is a real exception to “criteria first” and I want to name it clearly, because a rule with no exceptions is usually being oversold.

Reconciliation does not depend on criteria quality. Linking a pull request to a ticket, posting the commit evidence onto the ticket, and moving a ticket from in progress to in review based on the existence of an open review: none of these require the acceptance criteria to be any good at all. They require a link between a change and a ticket, which is a convention problem, not a specification problem.

So the honest version of the rule is narrower than the slogan. It is: anything that evaluates whether work is done, or gates on that evaluation, requires criteria first. Anything that merely records what demonstrably happened does not.

That distinction is practically useful because it lets you get the early, visible win of an accurate board while the slow criteria work proceeds in parallel. You do not have to hold the whole transition hostage to refinement practice. You just have to be disciplined about which side of the line each automation sits on, and honest that a ticket moving to done on merge sits on the evaluative side even though it feels mechanical.

How to make criteria work survive contact with a real sprint

Four tactics that I have seen actually work, as opposed to the ones that sound good in a process document.

Put the check at the exit of refinement, not the entry of the sprint. A ticket does not leave refinement until someone other than the author can state, out loud, how they would verify it. Ten seconds per ticket. If the answer requires a discussion, the ticket goes back, which is the correct outcome and is much cheaper now than in week two of the sprint.

Rewrite one ticket together, in the room, every refinement. Not all of them. One. It is the fastest way to transmit the standard, because standards transmit by example rather than by documentation. After four sessions the team writes them properly without the exercise.

Let the ambiguity surface rather than resolving it privately. When two people read the same criterion differently, that is the whole value of the practice being realised. The instinct is to smooth it over. Do not. That disagreement was always there; you have just found it before it cost you a sprint.

Make it visible when criteria quality decays, because it will. The single most common regression in this entire transition is that criteria improve during the push and drift back within a month. Whatever mechanism you use, scoring, a refinement checklist, a number in the retro, the point is that decay becomes noticeable rather than gradual and invisible.

What happens if you do it in the wrong order

Worth spelling out the specific failure modes, because “it goes badly” is not actionable.

The accurate-but-meaningless board. As in the opening. Status becomes reliable and the underlying definition remains contested, so the board is precise about an ambiguous quantity. This is worse than the drifting board, because a drifting board is visibly untrustworthy and this one is invisibly untrustworthy.

The gate everyone routes around. You enable a merge check against criteria that are not good enough to check against. It blocks legitimate work. Within two weeks, someone finds the override, and within three, the override is the normal path. You have now spent your one chance at introducing a gate.

The metric that eats the practice. Ticket throughput becomes measurable and accurate, so it becomes a number people are asked about, so tickets get sliced thinner. The criteria never improved, so nothing prevents a ticket from being defined as an arbitrarily small unit of work.

The blame transfer. Agents produce confidently wrong output because the criteria were vague. The vagueness is invisible; the wrong output is not. So the conclusion the team draws is that the agents are unreliable, and the actual defect, which was in the specification, never gets addressed. This one is common and it is the most expensive, because it produces a durable false belief.

Accurate but meaningless

Status becomes reliable while done stays contested. Precision about an ambiguous quantity, and it survives for years.

The gate everyone routes around

Enforcement against criteria too weak to enforce. The override becomes the path, and now you have neither.

The metric that eats the practice

Throughput becomes measurable, so it becomes a target, so tickets get sliced thinner and criteria never improve.

The blame transfer

Vague criteria produce wrong output. The vagueness is invisible and the output is not, so the agents get blamed.

Where this breaks down

I have stated this rule quite absolutely. Here is where it deserves qualification.

Some work legitimately resists machine-checkable criteria, and insisting anyway is harmful. Research spikes, exploratory prototypes, design-heavy work, incident investigation and anything whose purpose is to learn rather than to deliver do not decompose into verifiable outcomes without becoming absurd. Forcing them to produces fake criteria written to satisfy a process, which is worse than no criteria because it looks like compliance. The right answer is a category of work explicitly exempted from criteria requirements and correspondingly exempted from automated done-evaluation. If that category is more than roughly a quarter of your throughput, the ADLC framing may simply not be a good fit for your team, and that is a real conclusion rather than a failure of nerve.

“Criteria first” can become an excuse for never starting. I have watched teams use this argument to postpone indefinitely: we cannot automate until our criteria are good, our criteria are not good, therefore we do nothing. Six months later the criteria are the same and no reconciliation exists either. If you find yourself in that loop, the exception above is your escape hatch: turn on the purely recording automation, get the visible win, and let it buy you the political capital for the slow work. Sequencing is about which thing gates which, not about serialising everything.

And criteria quality has diminishing returns that arrive sooner than process enthusiasts admit. There is a point past which additional precision costs more than the ambiguity it removes. Criteria that specify implementation rather than outcome are worse, not better: they constrain the agent to your guess about how, they rot when the implementation changes, and they turn refinement into design-by-committee. The goal is evaluable, not exhaustive. A team that spends forty minutes per ticket in refinement has overcorrected, and the overcorrection will kill the practice faster than vagueness would have.

The takeaway

Automating an ambiguous process does not clarify it. It scales it, speeds it up, and gives it an audit trail that makes the ambiguity look intentional.

So: score criteria before you rewrite them, rewrite only what is both bad and imminent, build the habit at the exit of refinement rather than as a project, and hold the line that anything which evaluates or gates on done waits until the definition of done is checkable. Recording what demonstrably happened is the one thing you can safely do first, and you should, because it buys you time.

The next piece takes the same sequencing logic and applies it to the automation itself: which specific actions to enable first, which to enable much later, and which ones I think you should probably never automate at all.