Cost controls that do not degrade quality
Most cost controls are quality cuts in disguise, and the bill just moves to a line item nobody is watching. A few are free. Here is how to tell them apart.
A VP of engineering saw a model bill grow forty percent in a month and did the obvious thing. He halved the context window the team’s tooling was allowed to send, and he told everyone to use the smaller model unless they had a reason not to. The bill came down the following month, almost exactly as much as he expected, and he considered the matter closed.
What happened next took a quarter to become visible. Iteration counts went up, because the smaller model with less context needed more attempts. Review times went up, because reviewers were now catching things the tooling used to catch. And the defect rate on the affected repositories crept up enough that two incidents in the same quarter traced back to changes that had passed review in a hurry. The model bill was down. The cost of shipping a ticket was up.
This is the default outcome of cost control by blunt instrument, and it is worth understanding why, because the mechanism is not mysterious. He cut the cheapest input in the system to save money, and paid for it in the most expensive one.
Two families of control
Every cost control belongs to one of two families, and almost all of the argument about AI spend comes from failing to separate them.
The first family removes waste. Work the system was doing that produced no value: recomputing an identical answer, reading files that could not possibly matter, running an expensive check on something a cheap check already settled. Removing waste is free. It costs you nothing in quality because the removed work was not contributing any.
The second family removes work. Fewer checks, less context, a weaker model, a smaller sample. These controls do save money and they do reduce quality, in some ratio. That ratio might be excellent. Skipping a semantic review on a one-line typo fix is a fine trade. But it is a trade, and it needs to be made explicitly, with somebody watching the other side of the ledger.
The failure mode is applying a family-two control while telling yourself it is family one.
Removes work (a trade)
- Smaller model everywhere by default
- Hard token cap per request
- Check a sample of changes, not all
- Truncate context to fit a budget
- Disable verification on "small" diffs
Removes waste (free)
- Cache on an immutable key
- Send only what changed
- Cheap deterministic gate before the expensive one
- Skip generated and lock files
- Deduplicate identical questions in one run
The free ones, in order of payoff
Cache on something immutable. This is the largest and most consistently underexploited saving in agent tooling. If the input to an expensive computation is fully determined by a commit hash and a criterion, then the answer for that pair never changes, and the second time anyone asks, it should cost nothing. Re-runs are enormously common in practice: a flaky unrelated test, a rebase that changes nothing semantically, a reviewer clicking retry, CI re-triggering on a comment. Every one of those is a full recomputation unless you keyed a cache on the commit.
The discipline is choosing a key that cannot silently go stale. Commit hash plus criterion identifier plus tool version is safe. Branch name plus criterion is not, because branches move. Get the key wrong and you will serve a stale verdict, which is far worse than paying twice.
Send only what changed. An agent reasoning about whether a change satisfies a criterion needs the change, not the repository. The naive implementation sends far too much, and the cost of the call scales with what you send. Restricting to changed files is usually a large multiple of saving, not a marginal one, and it costs nothing in quality for the large majority of criteria. Where it does cost something, it is because a change’s effect is not local, which is a real case I will come back to.
Skip what cannot inform the answer. Lock files. Generated clients. Compiled assets. Snapshot fixtures. Minified bundles. These can dominate a diff by volume and contribute essentially nothing to whether the acceptance criteria are met. Filtering them is pure waste removal, and on some diffs it is the difference between a manageable payload and an enormous one.
Put a cheap deterministic gate in front of the expensive one. This is the single highest-leverage architectural decision in a verification pipeline, so it deserves more than a bullet.
The escalation ladder
Not every question needs a model. A great many acceptance criteria are checkbox-shaped: does this endpoint exist, is this field non-nullable, does this test file exist, was this migration written. Those can be answered by parsing, by static analysis, or by running the test suite, at a cost that rounds to zero.
The expensive semantic question, does this diff actually implement what this criterion describes, only needs to be asked about criteria the cheap layer cannot resolve. In a typical backlog that is a minority of criteria, which means the cost of semantic verification across your whole pipeline is a fraction of what a naive implementation would spend.
This is exactly how our own layer is built, and I mention it only because the shape generalises: deterministic checks first, semantic evaluation only on what remains unresolved, restricted to changed files with generated and lock files excluded, a cap on diff size, and results cached against the commit so a re-run is free. None of those five is a quality trade. Every one of them is removing work that was not contributing.
The diff size cap in that chain is the one control in the list that is arguably a trade rather than pure waste removal, and I want to be honest about it. A cap protects you from the pathological case, the pull request that touches nine hundred files because someone ran a formatter across the repository. It also means that genuinely enormous legitimate changes get degraded handling. The right response is not to remove the cap, it is to make exceeding the cap a visible event rather than a silent truncation, so that a human knows the change was not fully evaluated. A silent cap is a quality cut you have hidden from yourself.
The trades, and how to price them
Now the second family. These are legitimate, and I am not arguing against them. I am arguing that each one needs a matched metric on the other side.
Cheaper model for a class of work. Fine, and often correct. The metric that must be watched is iteration count for that class of work, because the usual failure is that the cheaper model needs more attempts and the total spend is flat while the wall-clock time got worse. If iterations hold steady, you found free money. If they rise, you moved the cost.
Sampling rather than full coverage. Checking a fraction of changes instead of all of them. This is defensible for advisory checks and indefensible for blocking gates, because a gate that fires on a random subset is not a gate, it is a lottery, and people will learn its odds. The metric is defect escape rate, and it moves slowly enough that you will not notice the degradation for a quarter.
Shorter context. The riskiest of the three, because its failure mode is silent and specific. The model does not tell you it lacked the file it needed. It produces a confident answer based on what it had. The metric is disagreement rate between the automated verdict and human review on the same change, which is worth measuring continuously anyway.
Turning off verification below a size threshold. Extremely tempting and worse than it looks. Small diffs are not safe diffs. A one-line change to a permission check, a flipped comparison operator, an off-by-one in a boundary, these are the classic sources of serious defects, and they all fit in three lines. Size is a poor proxy for risk. If you want a risk-based exemption, base it on the file path or the criticality of the module, not on line count.
The arithmetic that makes this worth doing
Let me put hypothetical numbers on the escalation ladder, purely to show the shape of the reasoning. These are invented. Actual prices move constantly and any figure I gave would be stale within a quarter, but the structure of the calculation survives price changes, which is the point.
Suppose a full semantic verification pass on an unscoped diff costs one unit. Suppose scoping to changed files and dropping generated files takes it to a quarter of a unit, because payload size is the dominant term. Suppose the deterministic gate resolves sixty percent of criteria at negligible cost, so only forty percent of criteria reach the semantic layer. And suppose thirty percent of verification requests are re-runs on an unchanged commit and therefore served from cache.
Start with a hundred units of naive spend. Scoping takes it to twenty-five. The deterministic gate takes it to ten. Caching takes it to seven. You are at roughly a fourteenth of the naive cost, and you have not weakened a single check. Every criterion is still evaluated, by the cheapest mechanism capable of evaluating it correctly.
Compare that to the VP at the top of this piece, who cut context in half and got a fifty percent saving that partly reversed itself in review time. The waste-removal path was an order of magnitude better and did not cost him anything downstream. That gap is why the distinction between the two families matters more than any individual technique.
Where this breaks down
The escalation ladder assumes the deterministic layer is trustworthy, and sometimes it is not. A cheap check that says a criterion is satisfied when it is not is far more damaging than no check, because it prevents the expensive check from ever running. The cheap layer must be conservative: it should only resolve a criterion when it can do so with near certainty, and it should escalate on ambiguity rather than guessing. Getting that bias wrong turns your best cost control into your worst quality problem, and the failure is invisible because everything reports green.
Changed-files-only has a real blind spot, and it is not hypothetical. Some changes have non-local effects. Deleting a function that seven other modules call. Altering a shared type. Changing a default in a configuration that a distant module reads. A verifier that sees only the diff will judge the diff correctly and miss the breakage entirely. Compilers and type checkers catch a lot of this, which is why the deterministic layer partly covers for the semantic one, but in dynamically typed codebases with loose coupling the gap is genuine. Be honest that scoping trades some correctness for a large cost saving, and that the trade is usually right without being free.
Caching has a subtler failure. If the tool that produces verdicts is upgraded, every cached verdict was produced by the old one. Including a tool version in the cache key fixes correctness and destroys your hit rate on upgrade day, which is fine and expected, but the first time it happens somebody will file a bug about a cost spike. Also, aggressive caching means a genuinely flaky verifier looks stable, because you froze the first answer it happened to give. That may be exactly what you want, but know that you chose it.
And the largest limit is that all of this optimises the wrong term. If you accept the argument in the first piece of this series, model spend is a small share of the cost of a ticket, and human attention plus rework dominate. Getting your inference bill down by a factor of ten is satisfying, and it is a rounding error against a review process that takes ninety minutes because nobody could tell what the ticket meant. Cost control on the model layer is worth doing because it is cheap to do and it removes a scaling risk. It is not where your money is.
The takeaway
Sort every proposed control into the two families before you implement it. If it removes work the system was doing for no reason, do it immediately and stop thinking about it. If it removes work that was contributing something, do it deliberately, name the metric on the other side, and watch that metric for at least a quarter, because degradation shows up slowly and the invoice improves immediately.
The free controls, in rough order of payoff: cache on an immutable key, scope the payload to what changed, drop what cannot inform the answer, and resolve everything you can with a cheap deterministic check before you spend on an expensive semantic one. Combined, these are usually worth an order of magnitude, which is more than any model swap will get you and does not cost you anything.
That leaves an obvious question hanging. Everything I just described, the ledger, the escalation ladder, the caching, the scoped payload, is a control layer, and somebody has to own it. The next piece is about whether you should build that yourself, which is a genuinely open question with a real answer on both sides.