Model choice as an economic decision
Teams pick models the way people pick text editors, on affinity and habit. It is a routing problem with a measurable answer, and the answer is different for different kinds of work in the same codebase.
Model selection in most engineering organisations is not a decision. It is a residue. Someone chose a tool eighteen months ago, the tool had a default, the default is still there. When the subject comes up, the conversation is conducted in the register of preference: this one is better at our stack, that one is more careful, the other one is chattier.
Some of that is real. Models do have characteristics. But the framing is wrong, because it treats the choice as a single global setting when it is actually a routing decision made many times a day, and because it treats quality as a property of the model rather than as a function of model, task and verification strength together.
The variables that actually matter
Three, and they interact.
Capability against your work. Not benchmark scores. How often the model produces a change that passes your criteria on the first attempt, on the kinds of tasks you actually have, in your codebase. This is measurable and almost nobody measures it.
Price per unit of work. Not price per token. A model at twice the price with a first-pass rate of eighty percent against fifty is cheaper, because failures pay the full context cost again and then consume human attention.
Latency, weighted by where it lands. A ninety second wait inside an interactive session costs an engineer’s attention and breaks flow. The same ninety seconds in a background pipeline costs nothing anyone notices. Latency is only expensive when a human is waiting.
Model A $1 per task, 50% first-pass expected attempts 2.0 -> $2.00 in tokens plus 1 human triage of a failed attemptModel B $2 per task, 80% first-pass expected attempts 1.25 -> $2.50 in tokens plus 0.25 human triage
Human triage at 15 minutes of an engineer’s time dominates both token figures by roughly an order of magnitude.
The general form: when human attention is in the loop, and it almost always is, the first-pass rate dominates the price per token. The cheap model is cheaper only when nobody has to look at what it produced.
Routing, not choosing
Once you accept that different tasks have different profiles, the global-setting framing collapses. Not every task deserves the same model, in the same way that not every change deserves the same review.
Mechanical and well-specified
Rename, format, boilerplate, an obvious test. Cheap and fast wins. Verification is strong, failure is visible and free.
Bounded but requires judgement
A typical ticket. Mid-tier, and this is where most volume sits. Worth measuring properly rather than assuming.
Novel or architectural
Best available, without much hesitation. The cost of a subtly wrong design is not measured in tokens.
Interactive and exploratory
Latency dominates. A fast adequate answer beats a slow excellent one when a person is waiting on it.
The routing signal does not need to be sophisticated to beat a fixed default. Change size, paths touched, whether the work is on a high-risk surface, whether a human is waiting: these are all available before the call is made, and a crude classifier over them captures most of the available value.
The place to be careful is the top-right quadrant. Downgrading architectural work to save money is the worst trade in the whole space, because the cost of a bad structural decision is paid over years by everyone who touches that code, and it is completely invisible at the moment you save the money.
The measurement nobody has
Everything above requires knowing your first-pass rate per model per task type, and almost no organisation does.
This is not because it is difficult. It requires attributing each attempt to a model and a piece of work, and recording whether it passed verification. If you already have a chain from intent to merge, the data is present and needs a query. If you do not, this is one more reason to build one.
The last row deserves separate treatment. A model with a high escape rate is dangerous in a way that no cost advantage compensates for, because its failures are the kind that reach customers. If two models differ on escape rate, that comparison ends the conversation regardless of price and first-pass performance.
Verification changes the calculus
The point most often missed: model choice and verification strength are substitutes, within limits.
With weak criteria, everything depends on the model getting it right unaided, because nothing downstream will notice if it does not. The pressure is to use the best available model everywhere, and it is correct pressure.
With strong criteria, a cheaper model’s failures are caught automatically and cheaply. Retry is a token cost rather than an incident. The economics shift substantially toward cheaper models on anything well-specified.
Which means investment in criteria has a return that shows up on the model bill, not just in quality. Teams evaluating whether better acceptance criteria are worth the effort usually score the quality benefit and miss this one entirely, and it can be the larger of the two.
The limit is worth stating: this only holds where correctness is expressible as criteria. Where it is not, verification cannot substitute for capability, and you are back to needing the model to be right on its own.
Where this breaks down
Measurement has a lag and the market moves faster. By the time you have four weeks of reliable data on a model, there is a new one. This is genuinely awkward and the practical response is to measure the task classification rather than only the model, so that when something new arrives you can evaluate it against a stable framework in days rather than starting over.
Multi-model operation has real overhead. Different failure modes, different rate limits, different prompt sensitivities, more integration surface, more to go wrong. For a small team, one good model everywhere is often correct even though it is provably not optimal.
Routing can be gamed by the people it routes. If engineers learn that small changes get the cheap model, some will split work to get the fast path, or inflate it to get the good one. Any classifier that people can see and influence will eventually be influenced.
First-pass rate is confounded by prompt quality. A model scoring poorly may be receiving worse instructions rather than performing worse. Comparisons need the same tasks and the same context assembly, and casual A/B comparisons frequently measure the harness rather than the model.
Provider commitments distort the decision. An organisation with a volume agreement has a strong pull toward one provider that has nothing to do with fit. That is a legitimate commercial reality and it should be named as such rather than rationalised into a technical argument, which is what usually happens.
The takeaway
Model choice is a routing problem, not a preference. The right model for a rename is not the right model for an architectural change, and the difference between them is worth more than any per-token negotiation.
Compare on cost per completed change rather than price per token, because the first-pass rate dominates once human attention is in the loop. Route on signals you already have. Never downgrade architectural work. And treat escape rate as disqualifying rather than as one factor among several.
If you take one thing into next week: work out your first-pass rate. One number, one model, one task type. Almost nobody has it, and it turns this decision from an argument about taste into arithmetic.
The next piece looks at the comparison that sounds flippant and turns out to be the most useful framing available: when the agent is cheaper than the meeting about the ticket.