Definition-of-done adherence as a first-class metric
For fifty years the definition of done was a document nobody read. It is now the interface an agent builds against, which makes adherence to it something you can and should measure per change.
Somewhere in almost every engineering organisation there is a wiki page called Definition of Done. It has a bulleted list on it. Tests written. Code reviewed. Documentation updated. Deployed to staging. It was written during a workshop, probably four years ago, and it is referenced approximately never.
Nobody is being negligent. That page was never load-bearing. The real definition of done lived in the head of whoever was building the thing, and it consisted of their understanding of what the requester actually wanted, refined by however many conversations they had along the way. The wiki page was a formality wrapped around a process that ran on human judgement.
Then we started handing tickets to builders that have no judgement, no context, and a strong tendency to do exactly what the text says. And the formality turned out to be the specification.
That is the change worth sitting with. The definition of done stopped being documentation and became an interface. Interfaces can be conformed to or not conformed to, and conformance can be checked. Which means, for the first time, adherence to the definition of done is a thing you can put a number on per change rather than a virtue you exhort people toward.
The two definitions of done
It helps to split the term, because it is doing two jobs and only one of them is new.
Process done is the wiki page. Tests exist, review happened, docs updated, staging deploy green. It is uniform across all work, it is largely mechanical, and honestly most of it has been enforced by CI for years. Nothing about agentic development changes it much, except that with more changes flowing the gates matter more.
Outcome done is whether the change actually does the thing that was asked for. It is specific to each piece of work, it lives in the acceptance criteria, and until very recently it was checkable only by a human reading the code against the ticket and forming an opinion.
Process done is table stakes and mostly solved. Outcome done is where the ADLC lives, and it is where the interesting measurement is, because outcome done is exactly what an agent can get wrong while satisfying every mechanical gate perfectly. Full test coverage of the wrong behaviour is a thing agents produce cheerfully.
So when I say definition-of-done adherence, I mean outcome done. Whether the diff satisfies the criteria the work was accepted on.
Process done
- The wiki page: tests, review, docs, staging green
- Uniform across all work
- Largely mechanical
- Mostly already solved
- Says nothing about whether it works
Outcome done
- Does it do the thing that was asked
- Specific to each piece of work
- Lives in the acceptance criteria
- Exactly what an agent can get wrong
- Where the interesting measurement is
Why this is measurable now and was not before
Comparing a code change against a paragraph of natural language and deciding whether one satisfies the other is a semantic judgement. For the entire history of software metrics that has been outside the reach of automation, which is why every metric we have counts events instead. Events are countable. Meanings were not.
That constraint has loosened. A model can read a diff and a set of acceptance criteria and produce a reasoned assessment of whether each criterion is met. It is not perfect and I will spend real time on the limits below. But it is a categorically new capability in the measurement toolkit, and it points directly at the gap that agentic development opened.
The design decisions around it matter more than the capability itself, and I have opinions on three of them.
It has to be per criterion, not per change. A single verdict on a pull request is close to useless, because the interesting output is not “does this pass” but “criterion three is not addressed anywhere in this diff.” That sentence is actionable in ten seconds. A score of seventy-two percent is not actionable at all.
It has to show its reasoning. An assessment that says a criterion is unmet without saying why is an assertion, and asserting things about people’s work without evidence is how automated systems lose the room permanently. The reasoning is the product. The verdict is a summary of it.
It has to have a threshold, not an average. Averaging adherence across a team produces a number that looks like a quality score and behaves like noise. What you want is a gate: below some confidence that all criteria are met, a human looks. The threshold is a policy decision about how much risk you are prepared to carry, and it should be set consciously rather than emerging from whatever the tool defaults to.
We landed on those three constraints in GroundTruth’s semantic acceptance-criteria checking for reasons that were mostly about not being wrong in public. An automated system that tells a developer their work is incomplete needs to be right, and where it cannot be sure, it needs to say so rather than guess.
What the metric actually tells you
Assume you have per-criterion adherence assessments flowing. What do you do with them?
The single-change use is obvious: it is a review aid. It tells the reviewer where to look, which is worth a great deal when the reviewer has fifteen pull requests waiting and used to have three.
The aggregate use is where it becomes a metric, and it is more interesting than a quality score. Three readings.
Adherence trend against throughput. If adherence is holding steady while merge volume climbs, your specification process is scaling. If adherence degrades as volume climbs, it is not, and you have found the constraint. This is the single most valuable thing the aggregate tells you.
Which criteria fail, not which teams fail. Cluster the unmet criteria. You will usually find patterns: criteria about error handling go unmet far more often than criteria about happy paths, or criteria written in one team’s style are unmet more often than another’s. Those patterns are about how requirements are written, which is fixable centrally, rather than about who is careless, which is not fixable at all.
Criteria that cannot be assessed. This is the sleeper. When the assessment cannot form a view on a criterion, that is usually because the criterion is not checkable: “the experience should feel responsive”, “handle errors gracefully”, “make sure it is performant”. The rate of unassessable criteria is a direct measure of how much of your specification is vibes. I would argue that number is more useful than the adherence rate itself, because it is measuring the input rather than the output, and the input is the thing you can change.
That last point deserves emphasis. The most valuable output of an adherence system is not a judgement about code. It is a judgement about requirements.
Adherence against throughput
Holding steady as volume climbs means specification is scaling. Degrading means it is not.
Which criteria fail
Cluster the unmet ones. Error handling fails far more often than happy paths, and that is a fixable pattern.
Criteria that cannot be assessed
The sleeper signal. Usually means the criterion was never checkable in the first place.
Not: a score per team
The one reading that turns a diagnostic into a stick and destroys the honesty it depends on.
The behaviour it should change
If this works, the visible change is not in engineering. It is in how work gets written up.
Acceptance criteria stop being a formality at the bottom of a ticket and start being the thing the ticket is for. They get written in a form that has a determinable answer: given this state, when this happens, then this is observable. They get reviewed before work starts, because a criterion nobody can assess is a criterion an agent will interpret creatively.
This is more work than writing “make checkout work”, and teams resist it, usually with the objection that it feels like a return to big up-front specification. I have some sympathy with the objection and I think it is wrong. Waterfall’s failure was specifying everything up front for a year. Writing three checkable criteria for a two-day piece of work is not that. It is defining the interface for a builder that has no ability to fill gaps from context, and it is required precisely because the builder is fast: a wrong interpretation now costs you a merged change rather than a conversation.
The metric matters because it makes the cost of vague criteria visible to the people writing them, which is the only mechanism I have ever seen actually change specification behaviour. Exhortation does not work. Showing someone that four of their last ten criteria could not be evaluated does.
Where this breaks down
This is the piece in the series where I am most conscious of arguing for something my own product does, so let me be harder than usual on it.
The assessor can be wrong, and confidently. A model reading a diff against criteria will sometimes mark a met criterion as unmet, and worse, will sometimes mark an unmet criterion as met because the code superficially addresses it. The first kind of error is annoying and self-correcting, because a developer will push back. The second kind is dangerous and silent, because nobody pushes back on being told they passed. Any adherence system will therefore have an asymmetric error profile that biases toward false confidence, and no threshold setting fixes that. Treat a pass as weak evidence and a fail as a prompt to look, never the reverse.
It cannot see what is missing from the criteria. Adherence measures the diff against the criteria. If the criteria omit something important, and they always do, a perfect adherence score tells you nothing about whether the right thing was built. This is the fundamental ceiling: the metric can verify conformance to a specification and can never verify that the specification was any good. Under the ADLC, specification quality is the actual constraint, and adherence only measures it obliquely, through the unassessable-criteria rate.
It pushes work toward what is checkable. This is the effect I worry about most. If criteria that cannot be assessed produce a visible penalty, teams will write criteria that can be assessed, and the easiest way to do that is to specify only mechanical, observable, low-ambition behaviour. Design quality, coherence, whether the feature is actually pleasant to use, none of these decompose into checkable criteria without becoming absurd. A team optimising for assessability will systematically underspecify exactly the parts of the work that most need human judgement, and the metric will improve while the product gets worse.
Some work legitimately resists it. Spikes, research, prototypes, refactors whose success criterion is “the next thing becomes easier”. Forcing acceptance criteria onto that work produces ritual rather than clarity. Any adherence programme needs an explicit exemption path, and the exemption needs to be socially acceptable to use, or people will write fake criteria to get through the gate.
Cost is real. Assessing every change semantically is not free, in money or latency. There is a threshold of team size and change volume below which the manual version, a human reading the ticket next to the diff, is simply better and cheaper. If a reviewer can hold the whole day’s changes in their head, they do not need this.
The takeaway
The definition of done went from a document nobody read to the interface a non-human builder constructs against. That promotion makes adherence to it measurable per change, and it makes vague acceptance criteria expensive in a way they never were before.
Measure it per criterion, with visible reasoning, against a threshold rather than as an average. Read the trend against throughput to see whether your specification process is scaling. And pay more attention to the rate of criteria that cannot be assessed than to the adherence rate itself, because that number is measuring your requirements rather than your code, and your requirements are the thing you can actually fix.
The next piece looks at drift, the widening gap between what your record says and what your repository shows, and why it turns out to be a leading indicator of schedule slippage.