Scoring your backlog's ability to define done
Before you fix anything, measure it. A concrete rubric for grading acceptance criteria at scale, and what the first run usually tells you.
The most useful hour I have ever spent on process was exporting a backlog to CSV and counting things.
Not reading it. Counting. How many tickets have anything in the acceptance criteria field at all. How many contain a digit. How many contain the word “properly”. How many have a checkbox. It takes about an hour with a spreadsheet and it produces a number that will change how your leadership talks about agent readiness, because it converts a vague anxiety (“our tickets are a bit thin”) into a fact (“sixty-one percent of our ready-for-dev tickets have no criteria that could be false”).
Everyone believes their backlog is mediocre. Almost nobody knows in which specific way, and the specific way determines what you should do about it, which is why this measurement is worth doing before any of the advice in the rest of this series.
Why measure before fixing
Three reasons, in ascending order of importance.
The obvious one is targeting. A backlog whose problem is missing criteria needs a different intervention than one whose criteria exist but are untestable prose. The first is a template and workflow problem. The second is a writing skill problem. If you guess wrong you will run a training session for a team whose real issue is that the field is not on the form.
The second is that quality is not uniform and the average will mislead you. In every backlog I have looked at, criteria quality correlates strongly with who wrote the ticket and moderately with which part of the product it touches. Two authors on the same team can differ enormously. An average tells you nothing actionable; a distribution tells you who to pair with whom.
The third and most important: you are about to spend real money on remediation, and you need a before number. Cleaning up a backlog is a multi-week effort that produces no visible feature output. It will get cut halfway through unless you can show movement. A score you can rerun weekly is the only thing that makes that work defensible internally.
The rubric
Here is the one I use. Six dimensions, each scored zero to two, for a total out of twelve per ticket. It is deliberately coarse. A finer rubric is more accurate and will not get used.
1. Presence. Are there acceptance criteria at all, distinct from the description?
- 0: none, or the field just restates the title
- 1: present but as a single sentence or a paragraph of prose
- 2: present as a discrete list of criteria
2. Falsifiability. Could an implementation demonstrably fail one of these?
- 0: no criterion could be shown to be violated (“works correctly”, “user-friendly”)
- 1: some criteria are falsifiable, at least one is not
- 2: every criterion could be shown false by a specific implementation
3. Concreteness. Are there numbers, literals, states, endpoints, or worked examples?
- 0: no digits, no literals, no named states
- 1: one or two concrete anchors
- 2: concrete anchors for every criterion that involves a quantity, an ordering, or a filter
4. Failure coverage. Is at least one non-happy path specified?
- 0: happy path only
- 1: errors mentioned generically (“handle errors”)
- 2: at least one specific failure case with its expected outcome
5. Machine readability. Is any of it in a form software can read the state of?
- 0: prose only
- 1: a list, but not checkboxes
- 2: checkboxes (markdown task list or native tracker task list)
6. Boundedness. Is the edge of the work defined?
- 0: contains “etc”, “and so on”, “as appropriate”, or an open-ended noun
- 1: mostly bounded, one soft edge
- 2: scope is closed, and ideally there is a line about what must not change
Bands: 0 to 3 is unusable by anything that cannot read your mind. 4 to 7 is workable by an experienced human who knows the domain. 8 to 10 is agent-ready. 11 to 12 is either excellent or over-specified, and you should read a few of them to find out which.
Automating the cheap two-thirds
You cannot hand-score four thousand tickets. You do not need to. Four of the six dimensions have decent mechanical proxies, and the proxies are good enough for a distribution even though they are wrong on individual tickets.
Export to CSV and run something like this. It is crude on purpose; the point is a number today, not a research project.
import csv, re
VIBE = re.compile(r'\b(work(s|ing)?|properly|correctly|user.?friendly|'
r'intuitive|seamless|clean|fast|performant|scalable|'
r'secure|gracefully|as appropriate|etc\.?|and so on|'
r'where relevant|if needed)\b', re.I)
CHECKBOX = re.compile(r'^\s*[-*]\s*\[[ xX]\]', re.M)
LISTITEM = re.compile(r'^\s*([-*]|\d+\.)\s+', re.M)
DIGIT = re.compile(r'\d')
LITERAL = re.compile(r'`[^`]+`|\b[45]\d{2}\b|/[a-z0-9_/-]+')
FAILURE = re.compile(r'\b(fails?|error|invalid|expired|rejected|denied|'
r'403|404|409|422|429|5\d\d|timeout|duplicate)\b', re.I)
def score(text):
t = text or ''
s = {}
s['presence'] = 2 if CHECKBOX.search(t) or LISTITEM.search(t) else (1 if len(t) > 40 else 0)
s['concrete'] = 2 if (DIGIT.search(t) and LITERAL.search(t)) else (1 if DIGIT.search(t) or LITERAL.search(t) else 0)
s['failure'] = 2 if FAILURE.search(t) else 0
s['machine'] = 2 if CHECKBOX.search(t) else (1 if LISTITEM.search(t) else 0)
s['bounded'] = 0 if re.search(r'etc\.?|and so on|as appropriate|where relevant', t, re.I) else 2
s['vibes'] = len(VIBE.findall(t))
return s
rows = list(csv.DictReader(open('backlog.csv')))
for r in rows:
r.update(score(r.get('Acceptance Criteria') or r.get('Description', '')))
n = len(rows)
print('tickets:', n)
print('no criteria at all:', sum(1 for r in rows if r['presence'] == 0) / n)
print('has checkboxes:', sum(1 for r in rows if r['machine'] == 2) / n)
print('has any digit or literal:', sum(1 for r in rows if r['concrete'] > 0) / n)
print('names a failure case:', sum(1 for r in rows if r['failure'] == 2) / n)
print('contains a vibe word:', sum(1 for r in rows if r['vibes'] > 0) / n)
print('mean vibe words per ticket:', sum(r['vibes'] for r in rows) / n)
Falsifiability and boundedness are the two that resist automation, so hand-score a random sample of forty tickets for those and extrapolate. Forty is enough to tell a disaster from a merely bad situation, and you can do forty in half an hour.
A word of caution on the vibe regex: work catches “workflow”, “workspace”, “working directory”. Either add exclusions or accept the noise, but do not present the number as precise. You want the order of magnitude, and the order of magnitude is usually shocking enough on its own.
There are tools that do this properly, including scoring criteria quality and flagging untestable language directly against your tracker rather than a CSV. Use one if you have one. The reason I am giving you the crude version is that the crude version can be run this afternoon without a procurement conversation, and the number it produces is what justifies everything after.
What the first run tends to show
I am not going to give you industry percentages, because I do not have data I would defend and the ones people quote are made up. What I will give you is the shape of the result, which is consistent enough to predict.
The bimodal distribution. You will not find a bell curve. You will find a cluster near zero and a cluster around seven or eight, with very little in between. Criteria quality is a habit, not a skill gradient: either the author writes criteria or they do not. This is good news, because habits transfer through pairing much faster than skills do.
The recency cliff. Tickets from the last two months score meaningfully better than tickets from a year ago, on backlogs that have had any process attention at all. Which means a chunk of your low scores are attached to work nobody intends to do. Filter by “touched in the last 90 days” before you panic.
Checkboxes are rare. Even teams with genuinely good criteria usually write them as bullets or prose rather than task lists. This is the single cheapest thing to fix in the whole exercise: same words, different syntax, and suddenly the state is machine-readable.
Failure coverage is the worst dimension, everywhere. It is normal for a large majority of tickets to specify only the happy path. If you only change one thing about how your team writes criteria, make it this one, because unspecified failure paths are where agent-authored code is least trustworthy and where the expensive incidents come from.
Epics score worse than stories, and bugs score best of all. Bugs score well because a bug report has a natural falsifiable structure: it did X, it should do Y. This is a hint about ticket templates that most teams miss.
Bimodal, not a bell curve
A cluster near zero and a cluster around seven or eight, with little between. Criteria quality is a habit, and habits are binary.
The recency cliff
Last two months score meaningfully better than a year ago. A chunk of the backlog is old, not bad.
Checkboxes are rare
Even good teams write bullets or prose. The single cheapest fix in the exercise.
Bugs score best
A bug report has a naturally falsifiable structure: it did X, it should do Y. That is a hint about ticket form.
Turning the score into a decision
The distribution maps to different interventions, and this is the actual payoff of measuring.
If presence is your weak dimension, this is a workflow problem, not a writing problem. The field is missing from the form, or it is optional, or nobody reviews readiness. Fix the template and add a definition-of-ready check. Cheap, fast, and it changes new tickets immediately.
If falsifiability is weak while presence is strong, you have a writing skill problem. People are trying. Run a session where the team rewrites ten of their own real criteria together, not a training deck. This is the intervention with the best return and the one most often skipped in favour of a document nobody reads.
If machine readability is the only weak dimension, congratulations, you have a formatting problem, which is the best problem. A template change plus a bulk edit of open tickets fixes it in a week.
If failure coverage is weak, add one required line to your template: “What should happen when this fails?” A prompt in the form outperforms a guideline in a wiki by an enormous margin.
If everything is weak and you have thousands of tickets, do not start at the top of the list. That is the subject of the next article, and the answer is triage rather than a march.
Rerun it weekly, on new tickets only
The metric that matters is not the backlog average. It is the score of tickets created this week, because that tells you whether the habit is changing. The backlog average moves slowly and is dominated by stale tickets, so it will make you feel like nothing is working even when the new work is transformed.
Track two lines: mean score of tickets created in the last seven days, and percentage of those with at least one checkbox and one named failure case. Put it somewhere the team sees it. That is the whole dashboard.
The metric that matters is not the backlog average. It is the score of tickets created this week, because that is the one that tells you whether the habit is changing.
And do not put it in anyone's performance review. The moment it becomes evaluated you will get twelve-out-of-twelve criteria that are precisely specified nonsense.
Do not put it in anyone’s performance review. The moment this becomes an evaluated metric you will get twelve-out-of-twelve criteria that are precisely specified nonsense, because the score is trivially gameable by an author who wants to game it. It works as a mirror and fails as a target.
Where this breaks down
The proxies are wrong on individual tickets, sometimes embarrassingly. A ticket reading “Change the button colour to #0B5FFF” scores badly on failure coverage and boundedness and is a perfectly good ticket. Small, obvious work does not need criteria, and a rubric that punishes it will push people to pad trivial tickets with ceremonial checkboxes. If you use this score at ticket level to block anything, you have misused it. It is a population instrument.
Score is not outcome. I have made an argument that better criteria produce better software, and I believe it, but the score measures the criteria and not the software. There is no controlled experiment here. A team could improve its mean from four to nine and ship exactly the same defect rate, and I could not prove otherwise from the numbers. Treat it as a leading indicator with a plausible mechanism, not as evidence of impact. If you want impact evidence, look at how often merged work needs rework, and whether tickets closed by automation get reopened.
It measures the artefact, not the understanding. Some of the best teams I have seen have thin tickets and extraordinary shared context, and they would score terribly. Their criteria live in people’s heads, which is exactly the thing that stops working when agents arrive, so the measurement is still telling you something real about agent readiness. But it is not telling you the team is bad, and if you present it that way you will lose the room in the first five minutes. Frame it as a readiness measure for a change in how work gets done, not as an audit.
And the exercise itself can become the work. There is a version of this where scoring, tracking and reporting on criteria quality consumes more effort than fixing the criteria would have. One hour to measure, one hour a week to track. If it is costing more, you have built a bureaucracy.
The takeaway
Measure before you remediate, because the shape of the problem determines the fix, and because you will need a before number to survive the remediation.
Six dimensions, scored zero to two: presence, falsifiability, concreteness, failure coverage, machine readability, boundedness. Automate four of them badly, hand-score forty tickets for the other two, and look at the distribution rather than the mean. Then track the score of newly created tickets weekly and nothing else.
You will almost certainly find that failure paths are your weakest dimension and that checkboxes are your cheapest win. Both are fixable in days.
The next piece is the hard part: what to actually do when the measurement comes back and you are staring at four thousand tickets that cannot define done, and marching through them in order is not an option.