Your test suite now certifies the bug
When the same interpretation produces the code and the tests, a wrong assumption does not just slip through. It gets a regression guard, a CI budget and a permanent defender. That is a liability with a recurring cost, and almost nobody accounts for it.
A test that encodes a misunderstanding is not a neutral artefact. It is a mechanism that will spend the rest of its life preventing anyone from fixing the misunderstanding.
That is the part of generated testing that I do not think has been priced properly. The familiar complaint is that a suite written from the same interpretation as the code cannot catch a specification error, which is true and well covered. The economic argument is harsher: the wrong test does not merely fail to help. It converts a one-time defect into a recurring cost, because from the moment it goes green it is the thing that will fail when someone tries to make the behaviour correct.
My position: stop counting tests as assets on the balance sheet. Every test has a carrying cost and a defence function, and a generated suite raises both at once. The teams that come out of this well will be the ones that treat test deletion as normal maintenance rather than as an admission of failure.
How a bug acquires a bodyguard
Trace the sequence, because the mechanism matters more than the sentiment.
A ticket says a refund is available within thirty days. The agent reads that, and reads it as thirty days from order date. The correct rule, known to two people in finance, is thirty days from delivery date. The agent writes the check, then writes the test, and the test asserts the behaviour the code has. It passes. It gets reviewed alongside two thousand other lines and merged.
Four months later someone in support works out the real rule. An engineer opens the code, fixes the date field, runs the suite, and gets a red build with a test named something like refund_window_expires_thirty_days_after_order. The test is specific, it is green on main, and it looks deliberate. The engineer now has to decide whether the test is wrong or their fix is wrong, with no evidence either way except a ticket from four months ago that says “thirty days”.
That fourth step is where the money goes. Not in the original mistake, which is cheap and inevitable, but in the friction of removing it later against an artefact that carries institutional authority.
Tests were evidence because they were expensive
For twenty years, the presence of a thorough suite told you something real, and it is worth being precise about what.
It told you a person had sat with the problem for long enough to enumerate cases. Writing tests was slow and tedious, so a large suite was proof of investment, and investment correlated with care. The suite was a proxy for attention, and the proxy held because the cost was real.
Generation removed the cost. A thousand-test suite is now producible in an afternoon by someone who has not read the domain rules. The artefact looks identical. The thing it used to be evidence of is simply absent.
This is the same pattern that has hit every effort-based proxy in software. When the cost of producing the signal collapses, the signal stops carrying information, and anyone still reading it as evidence is reading a number that no longer means what it meant. Developer trust in AI output has already fallen from around forty percent to twenty-nine percent in the Stack Overflow 2025 survey, which tells you the instinct is correcting even where the metrics have not.
The carrying cost nobody puts in the model
Here is the platform view, which is the one I am paid to hold.
Every test has an ongoing cost: CI minutes, wall-clock time on every pull request, flake investigation, and maintenance whenever the code it touches changes. In a suite of three hundred hand-written tests, that cost is roughly proportional to value, because a human made a decision about each one being worth writing.
In a suite of six thousand generated tests, the cost scales and the value does not. Most of them exercise paths no one cares about. A meaningful share assert implementation detail rather than behaviour, so they break on every refactor and get “fixed” by updating the expected value, which is a ritual, not a check. And some unknown number of them are certifying defects.
Test suite as asset
- More is better; count trends up and to the right
- Deleting a test needs justification
- Coverage percentage is reported to leadership
- Failures are always investigated as code problems
- CI cost treated as fixed overhead
Test suite as inventory
- Every test carries a runtime and maintenance cost
- Deleting a low-value test is routine housekeeping
- Reported signal is what the suite has caught, not its size
- A failure may mean the test is wrong; that is checked first
- CI spend is attributed and reviewed like any other line
What actually helps
Four things, in order of how much they return for what they cost.
Separate the source of the assertion from the source of the code. The assertion should come from the acceptance criteria, ideally written or confirmed by whoever owns the rule, before the implementation exists. This is the only fix that addresses the actual cause rather than the symptoms, and it is mostly a process change. Human-refined specifications have been shown in controlled studies to cut errors by up to half, and this is where that saving lives.
Record where each test’s expectation came from. A test whose assertion traces to a stated criterion is a different object from one an agent inferred. When CI goes red in month four, that provenance is the difference between an hour and a day. Most teams have nowhere to put this information, which is a tooling gap worth closing.
Spend your verification budget narrowly. Mutation testing on the ten percent of the codebase that touches money, permissions and data destruction will tell you more than any amount of breadth. It is expensive to run, so run it where being wrong is expensive.
Delete aggressively, and make it uncontroversial. A team that cannot delete tests will accumulate a suite that is slow, noisy and quietly wrong, and will keep paying for it forever. Give people explicit permission, and stop reporting test count anywhere a human might optimise for it.
Where this breaks down
A suite that certifies the current behaviour still has real value. Even if the behaviour is wrong, the tests tell you when it changes unintentionally, and unintentional change is a much more common source of incidents than an original misreading of a requirement. Ripping out generated tests because they might encode a misunderstanding would trade a known small risk for an unknown larger one.
“Delete aggressively” is dangerous advice at scale. In the wrong hands it becomes a licence to remove whatever is failing on a Friday afternoon. The version of this that works needs a criterion for deletion, ideally tied to what the test protects rather than to whether it is currently inconvenient. I do not have a clean rule for that, and anyone claiming they do is selling something.
Independence is achievable and I may be overstating the difficulty. Have one model or one person write the criteria, a separate process generate the implementation, and a third generate tests only from the criteria text. Teams doing this report it works. It costs more, it is fiddly to enforce, and it is not as independent as it looks, but it is a genuine answer and I should not pretend the problem is intractable.
Mutation testing on critical paths is expensive and unglamorous. It burns compute, it is slow, it produces findings that take judgement to triage, and it is very easy for it to become a report nobody reads. Recommending it is easy; sustaining it past the first quarter is where teams actually fail.
And test count as a metric has one honest use. A sudden drop can indicate someone disabled a suite to unblock a release. Removing the number entirely loses that. The fix is probably to keep it as an operational alarm and remove it from anything that looks like a performance review.
The takeaway
A generated test that encodes a wrong interpretation is worse than no test, because it converts a defect into a defended defect, and the defence has to be dismantled by someone with less context than whoever created it.
Treat the suite as inventory rather than as an asset. Know its carrying cost, know where its assertions came from, spend your real verification budget on the paths where being wrong is expensive, and make deletion normal.
If you take one thing into next week: take one failing test that somebody recently “fixed” by updating an expected value, and find out where that expectation originally came from. If nobody can tell you, you have just measured how much your suite is actually certifying.