Measuring ROI without lying to yourself
Almost every published productivity figure for agentic development is measuring something easy in place of something that matters. Here is what the honest version looks like, including the parts that are uncomfortable.
There is a genre of internal presentation that has become common enough to be predictable. It shows adoption climbing, a large percentage of code now agent-generated, and a productivity multiple derived from lines of code or pull request counts. It concludes that the investment has paid back several times over.
Almost all of these are wrong, and they are wrong in the same direction, and the people producing them are not being dishonest. They are measuring the things that are easy to measure, which happen to be the things that respond most readily to the intervention regardless of whether anything improved.
The result is a peculiar situation where the technology is genuinely valuable and most of the evidence offered for it is worthless. That is a bad position to be in, because when a sceptical CFO eventually pulls one of these numbers apart, the credibility loss lands on the technology rather than on the measurement.
The four measurements that fool people
Self-reported time saved deserves the harshest treatment because it is the most persuasive to executives and the least reliable. The research literature on self-reported productivity is unambiguous: people are poor at it, and the bias is not random. A survey asking “how much time did this save you” is measuring enthusiasm, and enthusiasm is real but it is not a return.
What honest measurement looks like
The defensible version measures outcomes at the boundary of the engineering organisation, not activity inside it.
Lead time from intent to production. Not from first commit, which excludes the part that is usually slowest. The clock starts when someone decides the thing should exist. This is the measurement that captures whether the bottleneck actually moved, and it is frequently the one that shows an uncomfortable result: generation got ten times faster and lead time barely moved, because generation was never the constraint.
That result is genuinely useful. It is the number that tells you to go and fix review capacity or environment contention instead of buying more agent seats.
Change failure and escape rate. Any productivity claim that does not report a quality number alongside it is incomplete, because trading quality for speed is trivially easy and superficially indistinguishable from an improvement. The correct form is a rate: defects per hundred changes, not defects in absolute terms, since absolute counts rise with volume even when quality improves.
Total cost per delivered change. Tokens plus tooling plus the human time in specification, review and rework. The human term dominates, which is why token-focused cost analysis misses the point in both directions.
Where work waits. Less a metric than a diagnostic, and probably the most actionable thing on the list. If a change spends four hours being generated and three days waiting for review, you have learned more from that than from any productivity multiple.
The counterfactual problem
Now the part that most treatments skip, and the reason honest measurement is genuinely hard rather than merely neglected.
To claim an improvement you need to know what would have happened otherwise, and you cannot observe that. The same period brought new hires, departures, a reorganisation, a quarter with less holiday, a platform migration finishing, and a team that was already getting faster. Attributing a delivery change to one intervention among those is not straightforward.
The available approaches are all imperfect and it is worth knowing how.
Weak evidence
- Before and after, no control
- Self-reported savings
- Activity counts of any kind
- Vendor benchmarks on other codebases
- A single team's enthusiastic pilot
Stronger evidence
- Staggered adoption across comparable teams
- Outcome metrics with a quality metric beside them
- Same-team comparison across task classes
- Sustained over two quarters, not one sprint
- Pre-registered: what would disconfirm this
Staggered adoption is the most practical of these and it costs almost nothing. If you are rolling out to twelve teams, roll out to four, then four more a month later, then the rest. You get a comparison group for free, and the comparison is between teams in the same organisation facing the same conditions, which controls for most of the confounders that make before-and-after useless.
Almost nobody does this, because the pressure is to roll out to everyone as fast as possible. That pressure is legitimate. It is also the reason the industry has very little credible evidence about a technology it has spent enormous sums on.
The last item on the right deserves its own sentence: decide in advance what result would make you conclude this is not working. If no possible outcome would change your mind, you are not measuring, you are documenting a decision already taken.
The J-curve, and why quarter one looks bad
One pattern worth expecting, because it causes good initiatives to be cancelled prematurely.
Adoption usually makes things worse before better. People are learning a new way of working. The criteria are not good yet, so verification is weak and review load is high. The pipeline is immature. Engineers are spending time on tooling instead of features. Measured honestly, the first quarter frequently shows a decline.
The organisations that get value are the ones that expected this and said so in advance. The ones that promised immediate returns spend the second quarter defending the programme instead of improving it, and a meaningful number cancel just before the curve turns.
Saying “this will look worse for a quarter” up front costs a little credibility at the start and buys a great deal of room later. It is also true, which helps.
Where this breaks down
Attribution stays imperfect no matter what you do. Staggered rollout is better than before-and-after and it is not a randomised trial. Teams are not interchangeable, adoption is not uniform, and people talk to each other across the boundary. Treat the result as evidence, not as proof.
Lead time is contaminated by things you do not control. A quarter with a security review backlog shows worse lead time regardless of how well the pipeline works. Interpret the number alongside the constraint diagnostic or it will mislead you.
Quality metrics lag. Defects surface weeks or months after the change that caused them, so a quality comparison over one sprint is measuring nothing. This asymmetry, where speed improvements appear immediately and quality regressions appear late, is exactly what makes early enthusiasm dangerous.
Some value is genuinely unmeasurable. Engineers spending less time on drudgery is worth something in retention and morale that no delivery metric captures. This is real and it is also the standard justification for every unmeasured claim ever made, so it should be stated as a belief rather than smuggled in as a benefit.
Measurement has its own cost, and it can be gamed. Building the pipeline to compute all of this is real work. And any metric that determines budget will eventually be optimised directly: publish a target for changes per engineer and you will get more, smaller changes within a month.
The takeaway
The easy measurements all move on day one regardless of whether anything got better, which is why they are the ones in every deck.
Measure lead time from intent, a defect rate alongside any speed claim, total cost per delivered change including human time, and where work actually waits. Stagger your rollout so you have something to compare against. Say in advance what would change your mind, and say in advance that the first quarter will look bad.
If you take one thing into next week: find out your lead time from intent to production, and compare it to your lead time from first commit. The gap between those two numbers is the part of your process that agents have not touched at all, and for most organisations it is the majority.
This closes the economics series. The remaining pieces step back from the operational detail and look at the longer arc, starting with an honest accounting of what the discipline gives up as it moves.