Writing acceptance criteria an agent cannot game
Precise criteria are not the same as unGameable criteria. Here are the specific patterns that look rigorous, pass cleanly, and deliver nothing.
A criterion I once thought was airtight:
- [ ] The CSV export includes all columns visible in the table.
Specific. Checkable. Names the artefact, names the scope, no weasel words. What came back was an export containing all columns visible in the table, where “visible” had been resolved against the default column configuration, because that is what the table shows when you load it. The three columns a user had toggled on were not in the file. The criterion was satisfied. The feature was useless to the person who asked for it.
Nobody cheated. That is the part worth sitting with. The implementation was a good-faith reading of the text, and the text had a hole in it that I could not see because when I wrote “visible” I was picturing a specific user with their specific columns turned on. The word was doing work in my head that it was not doing on the page.
This is the second-order problem with acceptance criteria in an agentic life cycle. Getting rid of “works correctly” is the easy half. The hard half is that criteria which look rigorous can still be satisfied by an implementation that misses the point entirely, and the more official your criteria look, the less anyone re-reads them.
Gaming is not malice, it is gradient descent
Let me kill the framing before it does damage. When I say “an agent cannot game this”, I do not mean the agent is adversarial. It is not trying to get away with something. It is doing the only sensible thing available: finding the shortest path to satisfying the text it was given, in a codebase with its own conventions, under a system prompt that rewards completion.
Every specification has a set of implementations that satisfy it. Good specifications have the property that the cheapest member of that set is also an acceptable member. Bad specifications have a cheap member that is garbage, and you will get that one, not because of intent but because it is nearer.
So the discipline is not “write more”. It is: read your criterion back and ask what the laziest possible thing that satisfies it looks like. If the answer is embarrassing, the criterion is not done. This is a five-second habit and it is most of the skill.
Every specification has a set of implementations that satisfy it. Good specifications have the property that the cheapest member of that set is also an acceptable member.
So the discipline is not writing more. It is reading the criterion back and asking what the laziest possible thing that satisfies it looks like. If the answer is embarrassing, the criterion is incomplete.
Pattern one: criteria satisfied by a stub
- [ ] Add a `/health` endpoint for the new service.
Laziest satisfying implementation: return 200. Which is, in fairness, what a lot of health endpoints do, and it will pass every check you have while telling you nothing about whether the service can reach its database.
- [ ] GET /health returns 200 with body {"status":"ok","db":"ok","queue":"ok"}
only when a SELECT 1 against the primary database and a ping to the
queue both succeed within 500ms.
- [ ] If either dependency fails or times out, /health returns 503 with the
failing dependency named in the body.
- [ ] An integration test covers the 503 path by pointing the service at a
closed database port.
The last line is the one that does the work. A criterion that only describes the happy path will be implemented as the happy path. If you do not name the failure case, the failure case does not exist as far as the specification is concerned, and code that has never once been executed with a failing dependency is code that does not handle a failing dependency.
Pattern two: criteria satisfied by the test being wrong
- [ ] Add unit tests for the discount calculator.
Laziest satisfying implementation: a test file containing three tests that assert the calculator returns what the calculator currently returns. Coverage goes up. The tests are tautological. They will pass forever, including after someone breaks the calculator, because they were derived from the implementation rather than from the intent.
This is the single most common way that “we require tests” produces no safety at all, and agents are extremely good at producing it, because generating a test from an implementation is a much easier task than deriving one from a specification.
- [ ] Discount calculator tests assert these exact cases:
- 0 items -> 0.00
- 3 items at 10.00 with no code -> 30.00
- 3 items at 10.00 with SAVE10 -> 27.00
- 3 items at 10.00 with SAVE10 and one item already on sale at 8.00
-> 25.20 (discount applies to sale price, not list price)
- expired code -> full price and a `code_expired` reason returned
- [ ] The sale-price case fails against the current implementation before
the change and passes after.
The values are the specification. Write the numbers down and the test cannot be derived from the code, because the numbers came from outside the code. The last line is a lightweight way of demanding that the test would have caught the bug: if it passes before your change, it is not testing your change.
Pattern three: criteria about the wrong side of the boundary
- [ ] The webhook handler validates the signature.
Laziest satisfying implementation: a function called validateSignature that is defined, is correct, and is called nowhere near the code path that matters. Or is called, and whose return value is not checked. I have seen both, from humans as well.
The criterion describes the existence of a behaviour rather than its effect at the boundary. Push it outward, to something observable from outside the unit:
- [ ] POST /webhooks/payments with a body whose HMAC does not match the
shared secret returns 401 and does not write to the `payments` table.
- [ ] POST with a valid HMAC but a timestamp older than 5 minutes returns
401 (replay protection).
- [ ] Replaying a previously accepted valid request returns 200 and results
in exactly one payment row, not two.
“Does not write to the table” is the phrase that makes this unGameable. It is an assertion about the state of the world, and there is no implementation that satisfies it while leaving the vulnerability open. Prefer criteria that assert a state change, or the absence of one, over criteria that assert a function exists.
Pattern four: criteria with an unbounded noun
Back to the CSV. The failure there was a noun, “visible”, that quietly referred to a context the specification did not carry. These are everywhere once you look:
- “all users” (active? including soft-deleted? including service accounts?)
- “the current period” (calendar month? billing period? the selection in the UI?)
- “recent activity” (how recent, and by what timestamp, created or updated?)
- “the report” (which one, and is it the cached one?)
The fix is not to define every noun in a glossary. It is to include one concrete instance in the criterion, so that any reading which does not cover that instance fails visibly:
- [ ] CSV export contains exactly the columns currently selected in the
table's column picker, in the same order, including columns the user
has toggled on that are not in the default set.
- [ ] Verified with the column picker set to: Name, Owner, Last seen,
Custom field "Region" (non-default). Export header row reads exactly
`Name,Owner,Last seen,Region`.
That second line takes fifteen seconds to write and it removes the entire class of failure. It is also the thing a reviewer can check in ten seconds without reading the diff. Worked examples inside criteria are underrated, and I now consider a criterion involving any collection, filter or ordering to be incomplete without one.
Pattern five: the conjunction that hides a decision
- [ ] Users can invite teammates and manage their permissions.
Two features in one box. When a checkbox covers two behaviours, ticking it asserts both, which means the box is either dishonest or blocking. Worse, the conjunction hides the interesting question, which in this case is what happens to a pending invitation when its permissions are changed before it is accepted.
Split them, and the hidden decision surfaces on its own:
- [ ] An admin can invite an email address at role `member` or `viewer`;
the invitee receives one email and appears in the members list as
`pending`.
- [ ] An admin can change a pending invitee's role before acceptance; the
accepted account gets the latest role, not the role at invite time.
- [ ] A non-admin calling the invite endpoint directly receives 403 and no
invitation is created.
The second criterion did not exist until the conjunction was broken. That is the usual outcome. Splitting compound criteria is not tidying, it is discovery.
Pattern six: negative space
Most criteria describe what the change should do. Almost none describe what it must not disturb, which is where agent-authored changes are least trustworthy, because an agent will happily refactor an adjacent module to make its own path cleaner.
- [ ] No change to the public response shape of GET /api/v2/orders; the
existing contract test passes unmodified.
- [ ] No new dependency added to package.json.
- [ ] Feature is behind flag `new_pricing`; with the flag off, behaviour is
byte-identical to current production for the fixtures in
`test/fixtures/pricing/*.json`.
The middle one looks petty. It is not. Unconstrained dependency addition is one of the most common and least reviewed things that happens in agent-authored diffs, and “no new dependency” is a criterion a machine can check trivially. The flag criterion is the strongest form of negative space available: it gives you a mechanical statement about the blast radius of a change, and it is checkable by running the old fixtures.
Two things that will not save you
Length. A criterion is not stronger for being longer. Some of the sharpest criteria I have written are eight words with a number in them. Some of the worst are paragraphs of prose that read like a design document and can be satisfied by anything. If you find yourself writing sentences with subordinate clauses, you are probably writing rationale rather than criteria, and rationale belongs in the description.
Formality. Given/When/Then does not by itself make a criterion checkable, which is a whole article on its own. “Given a user, When they use the feature, Then it works” is the same vibe with more scaffolding, and the scaffolding makes it harder to notice.
What actually helps is the boring stuff: a number, a literal string, a state assertion, a named failure case, a worked example. Those are the load-bearing elements. Everything else is presentation.
Will not save you
- Length. A criterion is not stronger for being longer
- Formality. Given/When/Then is not automatically checkable
- "Given a user, When they use it, Then it works"
- The same vibe with more indentation
Actually load-bearing
- A number
- A literal string
- A state assertion
- A named failure case
- One worked example
A checklist for the checklist
When I review criteria now, before any code exists, I run five questions. It takes about a minute per ticket.
- What is the laziest implementation that passes this? If it is acceptable, good. If not, the criterion is incomplete.
- Is there a number, a literal, or a concrete example anywhere? If a criterion mentions performance, size, count, ordering, or time and contains no digit, it is not finished.
- Which failure case is named? At least one, or you have specified the happy path only.
- What must not change? One line of negative space, usually.
- Can this be settled without a person? Not a requirement, but you should know the answer at authoring time rather than at merge time. Tooling can classify this for you (GroundTruth marks criteria it cannot mechanically settle as unverifiable and routes them to a human rather than guessing), but knowing it yourself while writing changes what you write.
Where this breaks down
Now the honest part, because everything above has a cost and I have watched teams pay too much of it.
Adversarial framing is mostly wrong and it can make you paranoid. I have written this article in the language of gaming and cheapest-satisfying-implementation because that framing generates good criteria. But it describes a tendency, not a reliable behaviour. Agents frequently exceed the specification, add the error handling you forgot, and write the test you did not ask for. If you write every criterion as though defending against a hostile contractor, you will produce criteria so constrained that they prevent the implementer from doing anything better than what you imagined. That is a real loss, and it gets larger as models get better. The correct posture is closer to “assume good faith and a literal reading” than “assume an adversary”.
Worked examples rot. A criterion pinned to Name,Owner,Last seen,Region is excellent on the day it is written and misleading eighteen months later when the column is renamed. Criteria with concrete literals in them are, in effect, a second test suite that nobody runs and nobody maintains. This is fine for the lifetime of a ticket and bad if you treat closed tickets as durable documentation. Either accept that criteria are disposable, or promote the ones that matter into actual tests and let the ticket text die.
Some of these patterns push toward integration tests that are expensive to run. “Does not write to the payments table” is a great assertion and it needs a database. Multiply that by every ticket and you have a CI suite that takes forty minutes, which produces its own pathology: people stop running it locally, batches get bigger, and feedback slows down. There is a real trade between assertion strength and cycle time, and the answer is not always “assert harder”.
And the biggest one: none of this catches a wrong feature. Every technique here makes the specification harder to satisfy accidentally. None of them make it more likely to be the right specification. You can write flawless, unGameable, mechanically checkable criteria for a feature nobody wants, and the process will run perfectly all the way to a useless outcome, faster than before and with better documentation. Criteria craft is a way of not losing information between intent and implementation. It has nothing to say about whether the intent was any good, and there is a failure mode where investing heavily in the machinery of specification makes teams feel rigorous while the actual product judgement quietly stops happening.
The takeaway
The goal is not detailed criteria. The goal is criteria whose cheapest satisfying implementation is one you would accept.
Five habits get you most of the way: name a failure case, put a number or a literal in it, assert a state change rather than the existence of a function, split conjunctions, and write one line of what must not change. Then read it back and ask what the laziest reading looks like.
Do that and you will find, as I did with the CSV export, that the holes were never in the parts you thought were vague. They were in the specific-sounding nouns you did not notice you were interpreting.
The next piece looks at Given/When/Then, which is the format most teams reach for when they decide to get serious about acceptance criteria, and asks what happens to it when the primary reader is no longer human.