Verifying criteria against the diff, and its limits
Reading a change and asking whether it satisfies a written criterion works better than expected, and fails in ways worth naming precisely before you rely on it.
Take a criterion written in plain prose:
Existing sessions are invalidated when a user completes a password reset.
And take a diff: forty lines across three files, one of which adds a call to sessions.revokeAllForUser(user.id) inside the reset completion handler, and a test that asserts an old cookie returns 401.
A person reading both takes maybe fifteen seconds to say yes, that is satisfied. There is no mystery in it. The criterion names a behaviour, the diff contains the behaviour, the mapping is direct. And this is a task that a model does well, because it is fundamentally a reading comprehension problem over two texts that are both present.
That is the whole idea behind semantic verification of acceptance criteria: instead of demanding that every criterion be a checkbox, read the change and ask whether the prose is satisfied. It closes the gap between what people naturally write and what a machine can act on.
It also has failure modes that are specific, predictable and worth naming before you put any weight on it, because the difference between a useful verification layer and a dangerous one is entirely in how it behaves when it does not know.
Why it works at all
The reason this is tractable is that a diff is unusually good evidence for a certain kind of claim.
A criterion about local, code-visible behaviour has its evidence concentrated in the change. “Returns 429 when rate limited” is answerable from the diff because the 429 either appears on the relevant path or it does not. “Validates the signature before writing to the database” is answerable because the ordering is visible. “No new dependencies” is answerable from one file.
These are not shallow keyword matches. Establishing that a revocation call is inside the reset handler and after the password update, rather than in an unrelated branch, requires actually following the code. But it is bounded reading over material that is entirely present, which is the regime where this kind of judgement is most reliable.
The second reason it works is that the question is narrow. It is not “is this code good” or “does this feature work”. It is “does this specific sentence describe something in this specific change”. Narrow questions with visible evidence are the good case.
The four things it is genuinely good at
Presence of a named behaviour. The example above. A criterion describing something that should exist in the code, checked against code that either has it or does not.
Absence assertions with a local scope. “No new dependencies”, “the public response shape of GET /orders is unchanged”, “no logging of the card object”. These are strong because the evidence for absence is bounded: you can read the whole diff and see that nothing added it. Absence claims about the whole system are a different matter, but absence claims about a change are checkable.
Test-coverage claims. “There is a test for the expired-token path” is answerable by reading the test file in the diff. This one is more valuable than it sounds, because it catches the very common case of a criterion being implemented without any test asserting it.
Contradictions. Sometimes the diff visibly violates a criterion, which is the highest-value output of the whole exercise. A criterion saying the flag defaults to off, and a diff setting it to true, is an unambiguous catch, and it is exactly the kind of thing a human reviewer skims past because the line looks like configuration boilerplate.
Notice all four are about the change itself. That is the pattern: evidence in the diff, claim about the diff. When the claim outruns the evidence, reliability falls off a cliff.
Where a confident answer would be a lie
- Runtime and scale: nothing in a diff answers p95
- System-wide absence: three files cannot prove "never anywhere"
- Data-dependent behaviour: needs the database
- Human experience: not close to answerable
- Cross-service: one service's diff is not enough
Where the evidence is in the change
- Presence of a named behaviour
- Absence assertions with local scope
- Test-coverage claims
- Contradictions, the highest-value output
Where a confident answer would be a lie
Anything about runtime or scale. “The dashboard returns in under 800ms at p95.” Nothing in a diff answers that. A reader can see whether the query looks efficient, which is a different claim, and the temptation is to answer the question that is answerable and report it as though it answered the one that was asked. That substitution is the single most dangerous behaviour in this space.
System-wide absence. “Passwords are never logged anywhere” cannot be established from a change that touches three files. The honest scope is “this change does not log passwords”. If a verification layer reports the broader claim as satisfied, it has silently changed the meaning of the criterion.
Behaviour that depends on data. “Existing users with no region set are handled” requires knowing what is in the database. A diff can show a null check, which is evidence, and cannot show whether the null check covers the actual distribution of legacy data. This is where I have seen the most plausible-but-wrong verdicts, because the code looks defensive and the reader has no way to know that the real legacy rows have an empty string rather than a null.
Anything about a person’s experience. “Feels lighter”, “is clear”, “reads well”. Not answerable, not close to answerable, and any confidence score attached to a verdict here is decoration.
Cross-service behaviour. In a system where the reset handler publishes an event and a separate consumer does the revocation, the diff for one service is not enough. This is a big one in practice, because the microservice case is common and the failure is silent: the diff looks incomplete, and a naive reader marks the criterion unmet, generating a false negative that erodes trust just as fast as a false positive.
The threshold is the whole design
Given all that, the design decision that determines whether such a system is an asset or a liability is not the model, the prompt, or the accuracy on the good cases. It is what happens below confidence.
A verification layer has three possible outputs and it must be willing to use the third. Met, not met, and needs human verification. The third one has to be a normal, frequent, unembarrassing outcome, because the alternative is a system that converts uncertainty into confident text, which is precisely the thing that makes a record untrustworthy.
This is how the semantic layer in GroundTruth is built: it runs at the edge, in CI, where the diff already is, and below its confidence threshold it returns needs-human rather than a verdict. The value is not that it settles every criterion. It is that it settles the mechanical ones honestly and declines the rest loudly enough that a person actually looks.
Two implications worth stating.
A high decline rate is a healthy sign, not a product deficiency. If a system claims to verify ninety-five percent of prose criteria with confidence, either your criteria are unusually mechanical or the system is bluffing. My prior is strongly on the second.
The threshold should differ by stakes. A criterion about a copy change and a criterion about authorisation should not clear the same bar. In the high-risk categories, the correct threshold is close to “always ask a human”, and the value of the automated pass there is not the verdict, it is the summary of what evidence was found.
What good output looks like
The verdict is the least useful part of the output. What a reviewer actually needs is the evidence, so they can check the reasoning in five seconds rather than re-deriving it in five minutes.
Criterion: "Existing sessions are invalidated when a user completes a
password reset."
Verdict: MET (high confidence)
Evidence:
- auth/reset.ts:88 calls sessions.revokeAllForUser(user.id) after the
password write, inside the same transaction
- test/auth/reset.spec.ts:140 asserts a pre-reset cookie returns 401
Not established:
- whether sessions held by the mobile client (separate token store)
are covered; no reference to mobile_sessions in this diff
The “not established” section is the most valuable part of that block. It is the difference between a verdict and a review. It gives a human the exact question to answer, and it is honest about scope in a way a bare MET never is.
And the declining case should be equally specific:
Criterion: "The import handles large files gracefully."
Verdict: NEEDS HUMAN VERIFICATION
Why: "gracefully" is not defined and no size threshold is given. The diff
adds streaming parse (import/parse.ts:34) which is consistent with large
file handling, but there is no test with a file above 2MB and no explicit
behaviour for memory or timeout limits.
Suggested criterion: "A 200MB CSV imports without exceeding 512MB RSS and
without a request timeout; a file above 500MB is rejected with 413."
That output is more useful than any verdict would have been, because it does three jobs: refuses to guess, explains the ambiguity, and proposes the fix. A verification layer that improves your criteria as a side effect of failing to verify them is doing something genuinely valuable.
The verdict is the least useful part of the output. What a reviewer needs is the evidence, and specifically the "not established" section, which gives them the exact question to answer.
The best output of the whole exercise is often not a verdict at all. It is a criterion rewritten to be checkable next time.
Running it at the edge
A practical note that matters more than it sounds. This check belongs in CI, next to the diff, not in a service that pulls your code somewhere else.
The reason is partly latency and cost, since the diff is already local and you are not shipping a repository across a network. But the bigger reason is that a lot of organisations cannot send source code to a third party, and a verification layer that requires it is simply unavailable to them regardless of quality. Running at the edge means the code stays where it is and only the verdict travels.
It also makes the check reproducible in the place where developers already look for verification results, which matters for adoption more than any feature.
Where this breaks down
Verdict quality is correlated with criterion quality, which is circular in an unhelpful way. Semantic verification works best on criteria that are already concrete, and criteria that are already concrete could mostly have been checkboxes. The criteria that most need help, the vague ones, are exactly the ones it must decline. So the honest description is not “this makes vague criteria checkable”. It is “this saves you from having to reformat good prose criteria into boxes, and it tells you which of your criteria are too vague to check”. Useful. Considerably less than the pitch.
False negatives cost more trust than false positives. A wrong MET is dangerous but invisible. A wrong NOT MET is visible, annoying, and lands on a developer who knows they are right. Three of those in a week and the team stops reading the output, which means the wrong METs stop being caught too. The engineering priority is therefore not accuracy in general but avoiding confident negatives, which usually means declining rather than asserting when the evidence is partial. That is an unusual optimisation target and it is easy to get backwards.
It reads the diff, not the codebase. Most of the interesting failures in software are interaction failures, and interactions are precisely what a diff does not show. A change can satisfy every criterion perfectly and break something two modules away. Verification against the diff is verification of intent-to-implementation fidelity. It has nothing to say about correctness in the large, and if the presence of green verdicts causes anyone to review less carefully, the net effect could be negative.
Non-determinism sits awkwardly in a gate. Model output varies. The same diff and criterion can plausibly return MET on one run and NEEDS HUMAN on another near the threshold. That is tolerable for an advisory signal and genuinely uncomfortable for anything blocking, because a rerun becomes a way to change the answer. If you gate on this at all, gate on the declines rather than the verdicts, and expect to explain the variance to someone eventually.
And there is a deskilling risk with a long fuse. Reviewers who see a criteria summary at the top of every PR will start reading the summary instead of the criteria. Over a year, the ability to hold “what was this supposed to do” in mind while reading a diff atrophies, and that ability is not recoverable quickly. I do not think this is a reason to avoid the tooling, but it is a reason to keep humans authoring criteria, which is where the understanding is actually built.
The takeaway
Checking prose criteria against a diff is a narrow, real capability. It works where the claim is about the change and the evidence is in the change: named behaviours, local absence assertions, test coverage, and outright contradictions. It cannot answer questions about runtime, scale, data distribution, cross-service flows, or human experience, and a system that answers them anyway is worse than nothing.
So judge these systems by one property: what they do when they do not know. Frequent, specific, well-explained declines are the sign of a tool you can trust. A high verified percentage is the sign of a tool that is guessing.
And the best output of the whole exercise is often not a verdict at all. It is a criterion rewritten to be checkable next time.
The next piece takes the decline seriously as a first-class outcome: when to say “needs human verification”, how to make sure that actually results in a human verifying something, and how the phrase turns into a rubber stamp if you are not careful.