Runbooks, metric definitions and the shared vocabulary problem
Two teams in the same company, using the same word, meaning different things. It was survivable when humans mediated. It is not survivable when agents write the query.
A finance team and a growth team once presented monthly active users to the same board meeting, forty minutes apart, with a difference of about nineteen percent. Neither number was wrong. Finance counted accounts with a billable event. Growth counted distinct authenticated sessions and included the free tier. Both definitions were defensible. Both had been in use for years. Neither team had any idea the other team’s number existed, because they worked in different dashboards and their paths had simply never crossed at the same altitude.
The meeting resolved it in the way these things always resolve: an uncomfortable ten minutes, an action item, and a decision to align on a definition, which produced a document that was never referenced again.
I have seen versions of this at every company above about fifty people. It is not a competence problem. It is a structural consequence of language: words drift toward the meaning that is locally useful, and teams are local.
What has changed is the cost.
Ambiguity used to be absorbed by humans
The reason this was survivable for decades is that a human sat in the middle of every use of an ambiguous term, and humans disambiguate automatically from context.
An analyst asked for “monthly active users” knows which team is asking and picks the right definition without noticing they made a choice. An engineer reading a ticket that says “suspend the account” knows that in this codebase suspension means something specific and different from what it means in the admin tool. That knowledge is not written anywhere. It lives as a kind of professional peripheral vision, acquired over months, and it silently corrects a hundred small ambiguities a day.
Take the human out of the middle and every one of those corrections stops happening.
An agent asked to add monthly active users to a report does not know there are two definitions. It finds one implementation, or it invents a reasonable one, and it produces a number with total confidence. The disambiguation step, which was invisible and free, has been removed, and nothing has replaced it. The ambiguity was always there. It was just being absorbed.
Human in the middle
- Disambiguates from context, unconsciously
- Notices when a term feels off
- Asks in a channel when unsure
- Ambiguity is absorbed, invisibly
- Cost is slow drift and occasional board embarrassment
Agent in the middle
- Picks the first plausible reading
- Has no sense of what is off
- Does not experience uncertainty as discomfort
- Ambiguity is resolved arbitrarily, at speed
- Cost is confidently wrong output, shipped
This is why the shared vocabulary problem, which everybody has always known about and nobody has ever prioritised, has become urgent. It moved from being a communication annoyance to being a correctness dependency.
A definition is a concept, and it has exactly one home
The Open Knowledge Format’s answer here is structurally simple and behaviourally demanding. A metric is a concept. It gets a file. The path is its identity. There is one file for monthly active users, and if there are genuinely two definitions, then there are two concepts with two paths and two names, and the fact that they are different is now visible in a directory listing rather than discoverable only in a board meeting.
That second half is the important part. The format does not resolve the disagreement. Nothing resolves the disagreement except people deciding. What the format does is make the disagreement impossible to hold accidentally, because two definitions cannot occupy one path.
That is a smaller claim than “single source of truth” and a much more honest one. Half the value of writing definitions down is not that everybody then agrees. It is that disagreement becomes a visible, discussable object rather than an ambient condition.
There is a naming discipline hiding in there worth stating plainly: if two teams cannot agree on a definition, do not give either of them the generic name. Give them both specific names and leave the generic term to point at both with an explanation. A term that means two things is a trap; two terms that mean one thing each are a vocabulary.
Runbooks are the other end of the spectrum
Metric definitions are declarative: they state what something is. Runbooks are procedural: they state what to do. They fail differently, and they are worth treating separately.
A runbook rots faster than almost any other knowledge, because it references the world at its most volatile: dashboard URLs, service names, the exact wording of an alert, which team owns escalation, which button is where. Every one of those changes on somebody else’s schedule.
And runbook rot has a nasty timing property. You find out the runbook is wrong at 3am, during an incident, when it is least recoverable, and the discovery costs you the thing you have least of, which is time and clarity.
The agentic version of this is worth thinking about carefully, because it is one of the places where automation is genuinely appealing and genuinely risky. A procedure that a human follows under pressure with judgement intact is not the same object as a procedure an agent executes literally. Give an agent a runbook that says “restart the ingestion workers if the queue depth exceeds ten thousand” and it will do exactly that, including in the case where the queue depth is high because of a deliberate backfill and restarting is the worst available action. The human read the same sentence and applied a hundred unwritten conditions.
Metric definitions
Declarative, slow-changing, high value per file. Rot is invisible and shows up in a number nobody can reconcile.
Runbooks
Procedural, fast-rotting, references volatile specifics. Rot is discovered at 3am.
Read by a human
Unwritten conditions applied automatically. Sentences are interpreted generously and sanely.
Read by an agent
Literal execution. Every unwritten condition is a missing condition, and every missing condition is a possible incident.
The practical consequence is that promoting runbooks into a machine-readable knowledge base raises the bar on how they are written. Every “obviously not in this case” that lived in the operator’s head has to become a written precondition, or the procedure has to be explicitly marked as human-only. Both are fine. Silently handing a human-written runbook to an agent is not.
The vocabulary is the interface
Step back and there is a general claim here that I think is underappreciated in discussions of agentic development.
Most of the effort in making agents useful goes into tools, context windows, orchestration and evaluation. Very little of it goes into vocabulary. But an agent operating inside your organisation is, functionally, a very fast new hire with perfect recall and no colleagues. The single biggest determinant of whether a new hire’s output is correct in month one is whether they use words the way your organisation uses them.
A knowledge directory of definitions is, in that framing, the onboarding document that never expires and never gets skipped. It is also the thing that makes agent output reviewable, because a reviewer can check the definition the agent used rather than trying to infer from the code what it believed.
This is where a control layer and a knowledge layer meet, and it is worth being precise about the boundary since they are often conflated. A knowledge base says what a word means. It cannot say whether a given change respected that meaning. That is a verification question, and verification has to happen against the actual change: the diff, the tests, the merge. GroundTruth’s position is that a claim about done has to be evaluated against ground truth rather than asserted, and the same split applies here. The knowledge base holds the definition. Something else has to check the work against it. Neither substitutes for the other.
Where this breaks down
Writing definitions down does not create agreement, and it can harden disagreement. Sometimes the reason two teams have two definitions is a live political disagreement about what the business should optimise, and writing both to disk makes the conflict legible without resolving it. That is usually progress. Occasionally it turns an ambient tension into a territorial fight over who owns the generic name. Be ready for that conversation rather than surprised by it.
Definitions are only as good as their connection to the implementation. A beautifully written file saying churn excludes trial accounts is worthless if the actual query does not exclude them. Now you have a documented lie, which is worse than an undocumented mess, because the documented lie is cited. Wherever possible, the definition file should link to the implementation, and ideally something should check that they agree, because a definition without a link to code is an aspiration.
Runbooks may not belong in an agent-readable knowledge base at all yet. I have argued they need explicit preconditions. The honest version is stronger: for high-consequence operational procedures, the unwritten conditions are numerous, situational, and partly unknowable, and the effort to write them all down may exceed the value. Marking a runbook as human-only is a legitimate answer and often the right one.
The maintenance burden lands hardest exactly here. Metric definitions and runbooks are the two categories most worth writing and the two most expensive to keep true. A directory of forty definitions is a standing commitment. If nobody owns it, it will be stale within a year, and stale definitions are the specific failure this article opened with, except now with the added authority of being written down and machine-read.
And the spec is v0.1. There is no defined vocabulary for what a metric definition should contain, no standard way to express supersession, and no notion of a verification link. Everything useful above is convention. That is fine, but it means “we adopted OKF” tells you almost nothing about whether a team’s definitions are trustworthy.
The takeaway
Shared vocabulary was always a problem and was always survivable, because a human sat between every ambiguous term and every consequential action, disambiguating for free. Agents removed that human, and the ambiguity that was being absorbed is now being resolved arbitrarily at speed.
One concept per file, with the path as identity, does not settle what a word should mean. It makes it impossible to disagree accidentally, which is most of the value. Name honestly when teams genuinely differ, link definitions to implementations, and be careful about handing human-written procedures to literal readers.
The next piece takes on the objection that has been sitting under this whole series: why bother curating structure at all when semantic search over your existing documents is one embedding job away.