QA: what changes when tests are also generated

Suppose you inherit a service and the first thing you do is look at its test suite. Nine hundred tests, all green, ninety-four percent line coverage. In 2019 that told you something real. Somebody cared. Somebody sat down and thought about edge cases. The suite was expensive to produce, and its existence was evidence of investment, which was itself evidence of quality.

Now suppose you learn that the code and the tests were produced in the same session by the same agent from the same ticket. Nine hundred tests, all green, ninety-four percent coverage. What do you know now?

Less than you think. Possibly much less. You know the code does what the code does. You know it does not crash on the paths the tests exercise. You do not know that it does what anyone wanted, because the assertions and the implementation came from a single interpretation of a single document, and if that interpretation was wrong then the tests are wrong in exactly the same direction as the code.

This is the quiet structural problem in agentic QA, and it does not have a clean fix. It has a set of partial fixes that are worth understanding properly, because a lot of teams are currently treating a green suite as though it means what it used to mean.

Tests were never checking the code

Here is the thing we all knew implicitly and rarely said. A test is not a check of the implementation. It is a check of the agreement between two independent expressions of the same intent.

A developer reads a requirement, forms a mental model, and writes code. Then they read the requirement again, form the mental model again, and write assertions. Those two acts are separated in time and often in mode of thinking, which is why writing tests so often makes you go back and fix the code: the second pass caught a divergence in the first.

The value came from the independence. Weak independence, admittedly. Same person, same day, same understanding. But not zero, and in practice enough to catch a meaningful class of mistakes. Test-driven development was a deliberate attempt to strengthen that independence by forcing the assertion to be written before the implementation existed to bias it.

Generated tests can have almost no independence. Same model, same context window, same interpretation of the same prose, often the same generation pass. Whatever the code got wrong about intent, the tests will confidently confirm. The suite becomes a very expensive tautology: this code behaves the way this code behaves, verified nine hundred times.

Coverage metrics make this worse rather than better, because coverage measures which lines executed, not whether any assertion meant anything. Ninety-four percent coverage with tautological assertions is ninety-four percent of nothing, and it looks identical on the dashboard to ninety-four percent of something.

Generated code plus generated tests

• Same model, same context, same prose
• Often the same generation pass
• Whatever the code misread, the test asserts
• Coverage rises, detection does not

Where the value always came from

• Two independent expressions of one intent
• Separated by time and by act
• Weak independence, but not zero
• Enough to catch a real class of mistakes

Restoring independence

If independence is the thing you lost, the useful question is how to get some back. There are several routes and they differ a lot in cost and strength.

Write the assertions before the implementation exists, from the requirement rather than the code. This is the strongest available answer and it is just TDD wearing new clothes. If acceptance criteria are written as named, decidable tests during specification (as argued in the previous piece in this series), then the tests exist before the code, they were derived from intent rather than from implementation, and the agent’s job is to make them pass rather than to describe what it built. The independence is real because the temporal ordering enforces it.

This is genuinely a lot of work up front. It is also the only approach here that addresses the root cause rather than sampling around it.

Separate the generation contexts. Have the tests written by a different pass with different inputs: give the test generator the requirement and the interface, not the implementation. This is weaker than it sounds (the same model with the same training will tend toward the same interpretation) but it is not nothing, and it is cheap. The failure mode to watch for is context leakage, where the test generator is handed the diff “for reference” and immediately collapses back into describing it.

Test properties rather than examples. Property-based testing has been a niche practice for two decades largely because writing good properties is hard and writing examples is easy. That calculus changes when generating examples is nearly free and therefore nearly worthless as evidence. A property such as “for any valid input, encoding then decoding returns the original” or “the ledger balance after any sequence of operations equals the sum of the entries” is a statement about intent that does not decompose into the implementation. Properties are much harder to write tautologically, because they are phrased in the domain rather than in the code.

Keep a small human-authored core. Not nine hundred tests. Twenty. The twenty that encode what this system is actually for, written by a person who understands the business, treated as protected files that agents may not modify. If a change breaks one of those twenty, that is a red alert rather than an inconvenience, and a human decides. This is cheap, it is easy to sell to a team, and it catches a surprising amount.

Mutation testing as the meta-check. If you want to know whether a suite has any teeth, deliberately introduce faults into the code and see whether the suite notices. Mutation testing was historically too slow and fiddly for most teams to bother with. It is now considerably more valuable, because it directly measures the thing you are worried about: does this suite actually assert anything. A generated suite with a low mutation score is a suite that is decorating your pipeline.

Assertions written first, from the requirementTDD wearing new clothes. The strongest answer and the only one addressing the root cause.

Separate the generation contextsGive the test generator the requirement and the interface, not the implementation.

Test properties, not examplesWriting good properties was always hard and writing examples always easy. That cost ratio just changed.

A small human-authored coreTwenty tests, not nine hundred. What the system is actually for, treated as protected.

Mutation testing as the meta-checkIntroduce faults deliberately and see whether the suite notices.

The rule that matters most

Out of all of the above, one operational rule does more work than the rest combined, and it is easy to state.

A change may not modify the tests that constrain it.

When a test fails there are two ways to make the pipeline green: fix the code, or fix the test. The second is faster, always available, and invisible in a green build.

So tests encoding acceptance criteria must be immutable within the change they constrain. If they need to change, that is a separate change with its own review.

When an agent implements a ticket and a test fails, there are two ways to make the pipeline green. Fix the code, or fix the test. The second is faster, always available, and completely invisible in a summary that says “all tests passing.” An agent optimising for a green pipeline with no stake in the outcome will take the cheap path, not out of malice but because the objective is underspecified in exactly that way.

So the tests that encode acceptance criteria for a piece of work must be immutable within that piece of work. If they need to change, that is a separate change, with its own review, and a human looks at it. This turns test modification from an invisible side effect into a visible, attributable event.

The practical implementation is unglamorous: a policy check on the diff that flags any change touching protected test paths, plus a review requirement on those specific files. Almost every team that has been burned by this has arrived at some version of it independently.

Once you have it, “the suite is green” recovers some of its old meaning, because green now implies the constraints were satisfied rather than negotiated.

Where QA effort actually goes now

Zoom out from the suite for a moment. If unit-level correctness is increasingly machine-produced and machine-checked, where should a QA function put its attention? Because the answer is not “nowhere,” and the framing of QA as a headcount to be reduced is a category error that will cost people real money.

Integration and interaction. Individual changes being correct says very little about six of them landing in the same afternoon. The failure mode in agentic teams is overwhelmingly at the seams: two changes that each satisfy their criteria and together produce a behaviour neither anticipated. This class of defect is under-tested at the best of times and dramatically under-tested when change volume goes up.

Adversarial and abuse testing. Agents build to the specification. Nobody specifies “does not leak other tenants’ data when the header is malformed” because it is obvious. Obvious is exactly the category that stops being handled. Deliberate adversarial testing, by humans who are thinking about what a hostile or careless user would do, is more valuable now than it was, not less.

Non-functional properties. Latency, memory, cost per request, database load. Agents optimise for the criteria they were given, and those criteria are almost always functional. A change that adds an N+1 query passes every functional test in the suite. Performance regression detection in CI moves from a nice-to-have to a primary defence.

Testing the specification itself. The newest and most important QA activity: reading acceptance criteria before implementation and asking whether they are decidable, bounded, and actually describe the desired outcome. A QA engineer who catches an ambiguous criterion has prevented a defect for a fraction of the cost of finding it later. This is the highest-leverage QA work available in an agentic team and most organisations have not created the role yet.

Notice that all four are judgement work at the boundaries of the system, and none is “run the regression suite.” That was always the part of QA that people found least interesting anyway.

Where this breaks down

Real limits, and one of them is bigger than the rest.

The independence argument may be weaker than I have made it sound. Human-written tests were also frequently tautological. Anyone who has reviewed a real-world suite has seen dozens of tests that assert the implementation back at itself, mocked so thoroughly that they verify nothing but the mock. The independence I am claiming we lost was never as strong as the ideal, and it is worth being careful not to romanticise the pre-agent baseline in order to make the argument tidier. The honest version: independence went from weak to nearly absent, and weak-to-none is still a real loss, but it is a smaller loss than “we had rigorous verification and now we do not.”

Writing tests first is expensive and most teams will not sustain it. TDD has been advocated for twenty-five years by extremely persuasive people and adoption has remained a minority practice throughout. Betting your quality strategy on the industry finally doing TDD because the incentives changed is a bet against a lot of historical evidence. The partial measures (protected core, mutation testing, separated contexts) are less effective and much more likely to actually happen, and a strategy that gets adopted beats one that does not.

Property-based testing has a hard learning curve. It is easy to recommend and difficult to do well. Bad properties are worse than no properties because they produce confusing failures that erode trust in the suite. A team adopting it needs someone who has done it before or a genuine tolerance for a slow start.

Mutation testing is expensive at scale. Running your suite hundreds of times against mutated code costs real compute and real wall-clock time. On a large codebase it is not something you run per commit. It becomes a periodic audit, which means it tells you about the state of your suite last month.

And the deepest limit: none of this addresses whether the requirement was right. Everything in this piece is about the fidelity of implementation to specification. If the specification described the wrong behaviour, a perfectly independent test suite will verify the wrong behaviour perfectly. QA in the agentic world drifts steadily toward being a specification-quality discipline, and the tooling for that is essentially nonexistent. We are quite good at checking that software does what it was told. We remain bad at checking that it was told the right thing, and I do not see that changing soon.

The takeaway

A green suite used to be evidence because two independent expressions of intent agreed. Generate both from one interpretation and the agreement becomes circular, while the dashboard looks exactly the same.

Recover independence where you can: assertions written from criteria before implementation exists, properties instead of examples, a small protected human-authored core, mutation testing to check whether the suite has teeth. And enforce the one rule that does the most work: a change may not modify the tests that constrain it.

Then move QA attention to where the defects actually are now, at the integration seams, in adversarial cases, in non-functional properties, and upstream in the quality of the specification itself.

All of this assumes the state of work is visible: which change satisfied which criteria, and when. Which brings up a ceremony built entirely around making state visible through human recall, performed daily, at fifteen minutes a head. The next piece asks whether the standup has any remaining purpose once the status it was designed to surface is derivable.