On-call for code nobody read
Your time to restore will look fine while your incidents get structurally worse, because the median hides what is happening in the tail. Putting someone on a rota for a system nobody has read is a staffing decision, not a documentation problem.
On-call was priced on an assumption that nobody ever wrote into the rota: whoever picks up the page has a fair chance of having context on whatever broke. Not certainty. A fair chance, backed by a colleague one message away who has a better one.
That assumption is being quietly withdrawn, and my position is that the response most organisations are reaching for, better runbooks, is aimed at the wrong thing. This is not a documentation gap. It is a staffing decision that nobody has consciously made, and your metrics are actively concealing it.
What the metrics will show you
Time to restore is a median in almost every organisation that reports it. Medians are the wrong statistic for this failure, and not slightly wrong.
Most incidents are ordinary. A deploy went out, something obvious broke, the graph points at it, you roll back. Those resolve fast, they dominate the distribution, and they are unaffected by whether anyone has read the code, because the diagnosis never required understanding anything.
The incidents where comprehension matters are the ones where the symptoms underdetermine the cause. There are more plausible explanations than time to test them, so the responder prunes, and pruning runs on a mental model. Take the model away and the search does not fail, it just takes much longer.
Those incidents are rare, so they sit in the tail. Your median absorbs them without moving. You can be getting structurally worse for three quarters with a flat, respectable time-to-restore chart, and the first honest signal will be a single very bad night that gets written up as bad luck.
What the dashboard reports
- Median time to restore, flat
- Incident count, flat or down
- Change failure rate, holding
- Deployment frequency, up
- Conclusion: the transition is going well
What to put next to it
- Ninetieth percentile time to restore
- Time from page to first correct hypothesis
- Share of incidents needing an escalation to find context
- Subsystems with no reader on the rota
- Conclusion: available, and rarely looked at
The single most useful of those is time from page to first correct hypothesis. It isolates the part of the incident that comprehension actually affects, and it separates cleanly from the mechanical part, the part where you deploy the fix and wait for it to propagate. Teams that start recording it usually find that it, not the remediation, is where their bad nights are spent.
Why runbooks do not close this
Runbooks are good, they are worth writing, and they answer a different question.
A runbook tells you what to do about a failure mode somebody anticipated. The incidents that hurt are the ones nobody anticipated, which is definitionally the set no runbook covers. What a responder needs at three in the morning is not a procedure. It is the ability to answer “why does it do that”, and the answer lives in a decision somebody made, months ago, for a reason that is not in the code.
Under the old arrangement, that reason lived in a person, and the person was usually still around. That is the asset going away, and it is not going away because people are leaving faster. It is going away because the number of decisions per person went up so far that no individual retains them, and a large share of the changes were merged under a service account with a one-line description.
So the replacement for a human memory is not prose. It is a record: this line changed under this ticket, with these criteria, approved by this person, on this date. That does not tell the responder what to do, and it collapses the search space in exactly the way the missing colleague used to, which is the whole job.
The part that is about people, not tooling
Here is the harder claim. Putting an engineer on a rota for a subsystem that nobody on that rota has read is not a documentation failure. It is asking a person to carry accountability for something you have not equipped them to understand, and that has consequences beyond the incident.
It produces the specific kind of on-call stress that drives good engineers out: not the interruption, which people tolerate, but the feeling of being responsible for outcomes you have no ability to influence. That is a retention problem, and it is arriving at the same time as the entry-level pipeline is narrowing, with postings for entry-level developer roles down around 40 percent. You are asking a smaller and more senior group to hold more surface area with less basis for holding it.
Two things follow, and both cost real money, which is why they get avoided.
Make reading a rota duty. Not an aspiration for a quiet sprint: named hours, on the schedule, for the person going on call next, spent on the subsystems they will be responsible for. It is the only intervention I know that directly addresses the missing input.
Track rota coverage as a staffing metric. For each subsystem you page on, how many people currently on the rota could explain its design. Where the answer is zero, you have not got a documentation gap, you have got an unstaffed system, and it should be escalated as one.
Where this breaks down
Comprehension may matter less than I think. A responder with good observability, distributed traces and a clean rollback path can resolve a great many incidents without understanding the code at all. Every hour spent on reading is an hour not spent on instrumentation, and instrumentation has a better-evidenced record.
Reading does not stay read. An engineer who spends four hours on a subsystem in January has a shallow model by June, and the code has moved underneath them. The half-life of the asset I am recommending you buy is short, and I have written about it as though it accumulates.
Assistance at incident time cuts the other way. An agent that can search the codebase and summarise unfamiliar subsystems in seconds is a genuine substitute for some of what the missing colleague provided, and it is getting better fast. If that trajectory continues, the comprehension gap becomes a tooling problem after all and my staffing argument ages badly.
Rota coverage is easy to satisfy dishonestly. The moment “who could explain this subsystem” becomes a reported number, names will appear next to subsystems on the basis of a skim. I have proposed a metric with an obvious cheat and no defence against it beyond the integrity of the person filling it in.
And the tail may just be the price. If volume delivers enough value, an occasional very bad night could be a rational trade, made deliberately. My argument is that nobody is making it deliberately. Someone who has made it with open eyes and priced the tail is not doing anything wrong.
The takeaway
The comprehension your on-call rota depended on was never in a document. It was in people, it is thinning, and your median time to restore is built to hide that.
Measure the tail and the time to first correct hypothesis. Give responders a record that collapses the search the way a colleague used to. And treat a subsystem with no readers on the rota as an unstaffed system rather than an undocumented one.
If you take one thing into next week: list the subsystems that page you, and next to each write the number of people currently on the rota who could explain its design. Every zero on that list is a decision you have made without noticing.