Checkbox criteria versus semantic criteria
One has a state a machine can read. The other needs judgement to settle. Most teams mix them without noticing, and then wonder why automation is unreliable.
Two tickets, both marked ready, both with acceptance criteria, sitting next to each other in the same sprint.
Ticket A:
- [ ] Rate limit of 100 requests/minute per API key on POST /v1/messages
- [ ] Requests over the limit return 429 with a `Retry-After` header
- [ ] Limit is per key, not per IP
Ticket B:
The new onboarding flow should feel lighter than the current one and
guide the user to their first successful send without overwhelming them
with configuration options up front.
Both are legitimate. Ticket B is not badly written; somebody thought hard about it, and it communicates real intent that a checkbox would flatten. But they are different kinds of object, and the difference is not quality. It is whether anything other than a person can tell you if they are satisfied.
Ticket A has a state. A machine can look at those three boxes, see whether they are ticked, and act on the answer. Ticket B has a meaning. There is no operation that resolves “feels lighter” to true or false, and any system that claims to have performed one is bluffing.
Most teams have both kinds in the same field, unlabelled, and then build automation on top as though they were the same thing. That is where the trouble starts.
Semantic criterion
- Carries meaning and intent
- Shapes a hundred small decisions
- No state a machine can read
- Settled only by a person
- Flattened if forced into a box
Checkbox criterion
- Carries a state, not a meaning
- Says nothing about why
- Readable and actionable by software
- Settled by evidence
- Becomes a lie if ticked as ritual
What a checkbox actually buys you
A checkbox is not better writing. It is a different data type.
When criteria are written as a markdown task list, or as a native task list in your tracker, they become the only part of a ticket description with a machine-readable state. That has a specific consequence at merge time: if a pull request lands and two boxes are unticked, that is a fact, not an inference. Software can act on facts without guessing. The correct action is not to force the ticket to done and not to leave it silently in progress, but to move it to review, which says exactly what is true: the code landed, the definition of done is not fully met, a person should look.
That single behaviour resolves most of the drift that makes boards untrustworthy. It is unglamorous. It is also close to the whole value proposition of mechanical criteria, and it depends entirely on the criteria having a state to read.
The costs are real too. A checkbox flattens nuance. It invites the ticking of boxes as a ritual rather than an assertion, and a ticked box that nobody verified is worse than no box, because it manufactures false confidence and leaves a record saying someone confirmed it. And plenty of important properties genuinely do not decompose into binaries without becoming silly.
What semantic criteria buy you
Ticket B carries something the checkboxes cannot. It tells the implementer what the feature is for. A checklist derived from it might say “no more than 3 fields on the first screen” and “the send button is reachable in 2 clicks”, and those are useful, and they are also not the same as understanding that the point is to get someone to a first success quickly.
That understanding matters more now, not less. An implementer with the intent can make a hundred small decisions in the right direction. An implementer with only the checklist makes them in whatever direction is nearest. The prose is doing real work.
So the answer is not “convert everything to checkboxes”. It is to be clear about which artefact is doing which job, and never to let a system pretend it has settled something it has only read.
The three honest states
The useful mental model is that every criterion, at merge time, resolves to one of three things.
Met. Something checked it. A ticked box, a passing test named in the criterion, a CI job that asserts the condition. There is evidence.
Not met. Something checked it and it failed, or the box is empty. Also evidence, of the more useful kind.
Unverifiable. Nothing available can settle it. Not “probably fine”, not “assumed met”. Unverifiable is a real answer and it is the one that most needs to survive intact, because the whole failure mode of automated definition-of-done is a system quietly converting unverifiable into met to keep the pipeline moving.
Text criteria and Given/When/Then land in the third bucket by default. That is not a criticism of them. It is a description of what they are. A system that treats them honestly routes them to a human and says why. A system that guesses at them buys you speed and sells you your ability to trust the board, which is a bad trade, because a slow honest record beats a fast dishonest one every time.
The semantic layer, and what it can honestly claim
There is a middle path that has become practical: read the actual diff and compare it against the text criterion, using a model, and produce a judgement with a confidence estimate.
This works better than you might expect for a specific class of criterion. “The endpoint returns 429 when rate limited” is prose, but the diff either contains a 429 path or it does not, and a model reading both can say so with reasonable confidence. “Existing sessions are invalidated on password reset” is checkable against a diff in the same way. These are prose criteria about concrete, local, code-visible behaviour, and the evidence needed to settle them is right there in the change.
It works badly, and should be honest about working badly, in three situations. When the criterion is about a property of the whole system rather than the change (“the dashboard is fast”). When it depends on runtime behaviour or data not present in the diff. And when it is about human experience, which is Ticket B, where no amount of diff reading substitutes for a person opening the thing.
Reads well against a diff
Concrete, local, code-visible behaviour. "Returns 429 when rate limited." The evidence is in the change.
Whole-system properties
"The dashboard is fast." Nothing in one diff settles a property of everything.
Runtime and data dependent
Behaviour that only appears against real traffic or real records. Not present in the change to be read.
Human experience
Ticket B. No amount of diff reading substitutes for a person opening the thing.
The design decision that makes this trustworthy rather than corrosive is the threshold. Below some confidence, the answer has to fall back to “needs human verification” rather than to a guess. GroundTruth’s semantic layer runs this at the edge in CI, where the diff already lives, and drops to needs-human below its threshold rather than producing a confident-sounding verdict. The value is not that it settles everything. It is that it settles the easy ones honestly and declines the rest loudly.
If you take one thing from this article: a verification system’s credibility is set by what it does when it does not know. Everything else is table stakes.
Writing both, deliberately
The practical technique is to stop treating the acceptance criteria field as one homogeneous blob and instead write it in two clearly separated parts. Here is a full example for the onboarding ticket, which is the hard case because its intent is genuinely qualitative.
## Intent
New users are dropping out before their first successful send. The flow
should get someone from signup to one delivered message with the minimum
configuration, and defer everything that can be defaulted. If a decision
can be made for the user now and changed later, make it for them.
## Mechanical criteria
- [ ] The first screen after signup collects at most 2 fields (name,
use case) and no API configuration
- [ ] A default project and default API key are created automatically at
signup; the user is never asked to create either
- [ ] The "send a test message" action is reachable in 2 clicks or fewer
from the post-signup screen
- [ ] A user who abandons at any step and returns lands on the furthest
step they completed, not step 1
- [ ] Existing users are unaffected: the old flow renders for accounts
created before the flag date, verified by the fixtures in
`test/fixtures/onboarding/legacy_*.json`
- [ ] Instrumentation emits `onboarding_step_viewed` and
`onboarding_first_send` with the account id
## Needs human verification
- Does the flow read as lighter, or just shorter? Product review with
two people who have not seen the designs.
- Copy on the empty state: sign-off from whoever owns tone.
- Mobile web at 375px: does the two-field screen still feel like a form
rather than a wall?
Three sections, three different contracts. The intent section is for the implementer’s judgement, and it is the part that makes the mechanical criteria make sense. The mechanical section can gate a merge. The third section is not a failure of specification; it is a specification of what the machine must not pretend to know, and naming it means the review actually happens instead of being skipped because everything looked green.
I would rather work from that ticket than from either half alone, and it is not much longer than the prose version.
Do not fake the conversion
A tempting move once you know that checkboxes are mechanically preferable is to convert everything, which produces criteria like:
- [ ] The onboarding flow feels lighter
This is the worst artefact in this entire article. It has the data type of a checkable criterion and the content of a vibe, so it will be ticked by whoever is closing the ticket, and now your system has a record asserting that lightness was verified. You have not made anything checkable. You have laundered an opinion into evidence.
The same goes for - [ ] Code reviewed, - [ ] Tested, and - [ ] Works as expected, which appear in an enormous number of ticket templates and are pure ceremony. If a box will always be ticked, delete it. Its only function is to add noise to the signal you are trying to build.
Where this breaks down
The three-section format is more structure than most teams will sustain. It works when the ticket is meaningful and someone owns the writing. On a board with two hundred small tickets, the overhead per ticket is not worth it, and a team that tries to apply this uniformly will apply it badly and then blame the format. The realistic version is: use it on the tickets that are genuinely ambiguous or genuinely risky, and let the rest be three checkboxes.
Checkbox state is only as good as the honesty of whoever ticks it. Everything in this article rests on a ticked box meaning something. In practice boxes get ticked in bulk at the end of a sprint, sometimes by someone who did not do the work, occasionally by an agent instructed to close out its own ticket. The data type is machine-readable; the truth value is still socially produced. A team with a box-ticking culture will get exactly the same drift as before, now with an audit trail that says otherwise. That is arguably worse, and no tooling fixes it.
The mechanical/semantic split can become a dumping ground. Once “needs human verification” exists as a category, it is very easy to put anything hard in it and move on. If your ratio drifts toward mostly-human, you have not built a definition of done, you have built a to-do list for a reviewer who does not have time. I would treat a rising unverifiable ratio as a signal that criteria authoring has degraded, not that the work got harder.
And semantic verification has a subtler risk than being wrong. It can be right often enough that people stop reading the cases where it declines. A system that says “needs human verification” on ten percent of criteria, and is trusted on the other ninety, trains reviewers to skim. The declines are the important output, and they arrive looking like noise. I do not have a clean answer to this beyond keeping the decline rate visible as a number people watch.
The takeaway
Checkbox criteria and semantic criteria are not better and worse versions of the same thing. One has a state a machine can read and act on; the other has a meaning that shapes judgement. You need both, and you need them separated, because the damage happens when a system treats the second kind as though it were the first.
Write the intent as prose. Write the gate as boxes. Write the parts that need a person as a list of things a person must actually do. And insist that any automation you adopt says “I do not know” clearly and often, because that is the only property that makes the rest of it trustworthy.
The next piece gets empirical: how to score your existing backlog’s ability to define done, what to measure, and what the numbers usually look like the first time anyone runs them.