Start in shadow mode, proving before writing

There is a moment in every one of these rollouts where somebody says, reasonably, “we have seen the demo, the accuracy is fine, can we just turn it on?”

It is a fair question and the answer is no, and the reason is not caution for its own sake. It is that the first time automation touches a team’s board, you spend a one-time budget of trust that you cannot re-earn cheaply. If the first three actions the team sees are right, you have a rollout. If one of the first three is visibly, stupidly wrong, you have a story that will be repeated in that team for eighteen months, and no amount of subsequent accuracy will overwrite it. People remember the ticket that got closed while they were still working on it. They do not remember the four hundred that were moved correctly.

Shadow mode exists to make sure the first three actions the team sees are right.

What shadow mode actually is

Precisely: the system connects to your repositories and your tracker, reads everything, detects every event it would normally act on, decides what it would do, records that decision along with the confidence behind it, and then does nothing at all. Zero writes. No status transitions, no comments, no links, no notifications. From the perspective of anyone on the team, nothing has been installed.

What you get is a log. Every entry says: here is what I saw, here is what I would have done, here is how sure I was.

That log is the artefact. Not the tool, not the integration, not the dashboard. The log is what turns a conversation about a vendor’s claimed accuracy into a conversation about your own codebase’s measured accuracy, and those are completely different conversations. One of them you can win.

event    PR #4412 merged, branch feat/tenant-quota
saw      no ticket reference in branch, title or body
would    link to PLAT-2210 (files overlap, author, window)
confid.  0.71
action   NONE (shadow)
event    PR #4418 merged, branch PLAT-2231-retry-backoff
would    transition PLAT-2231 -> code complete
confid.  0.96
action   NONE (shadow)

Why it is worth a fortnight

The argument against shadow mode is time. Two weeks, or better a full sprint or two, in which nothing improves and nobody saves any labour. In a quarter that is a meaningful chunk.

Here is what you buy with it.

You find your own error modes, not the generic ones. Every codebase has idiosyncrasies that break inference. A team that puts ticket keys in commit messages but not branch names. A monorepo where one pull request routinely satisfies parts of four tickets. A convention where a ticket is closed only after a separate QA sign-off that lives outside both systems. None of these are exotic and none of them show up in a demo. All of them show up in your shadow log in the first week.

You get a number you can defend. After two weeks you can say: across two hundred and forty detected events, we agreed with the proposed action in this many cases, and here are the categories of the ones we did not. That sentence ends arguments. A vendor’s benchmark does not.

You calibrate the threshold on evidence. Every one of these systems has a confidence threshold: above it, act automatically; below it, route to a human. Choosing that number in the abstract is guesswork. Choosing it after looking at two weeks of your own scored decisions is engineering. You can literally sort the log by confidence and find the point below which the errors cluster.

You de-risk the political problem, which is the real problem. The technical failure mode of this transition is a wrong ticket transition, which is annoying and revertible. The organisational failure mode is a team that decides the automation is untrustworthy in week one and quietly routes around it forever. Shadow mode is the only tool that addresses the second one, because it lets you be wrong in private.

How to run it properly

Shadow mode run badly is just a log nobody reads. Four things make the difference.

Run it for a full cycle, not an arbitrary two weeks. If your team works in two-week sprints, one sprint is the floor and two is better, because a sprint has a shape: refinement at the start, a merge rush at the end, and the end is where the interesting failures live. If you have a monthly release train, span a release. The unit is a cycle of your team’s actual rhythm, not a fixed number of days.

Review it weekly, with the tech lead, for fifteen minutes. Not the engineering manager, not a programme manager. The person who knows what the branches mean. Scroll the log, ask “would we have wanted that?”, and mark disagreements. Fifteen minutes a week for two weeks is the entire human cost of shadow mode, and teams that skip the review get all of the delay and none of the benefit.

Categorise disagreements rather than counting them. A raw error rate is much less useful than a taxonomy. In practice, disagreements almost always sort into a handful of buckets:

• Convention gaps. The system could not link a change because the reference was missing or malformed. Fix the convention, not the tool.
• Granularity mismatches. One ticket describes several changes, or one change satisfies several tickets. This is a refinement problem wearing a technical costume.
• Out-of-band process. Work is genuinely not done at merge because something happens elsewhere: a manual QA gate, a staged rollout, a compliance sign-off. This is real and it means your rules need to reflect it.
• Genuine inference errors. The evidence was there and the conclusion was wrong. These matter most and, in my experience, they are the smallest bucket by a wide margin.

The reason to categorise is that only the last bucket is a case for distrusting the system. The first three are cases for changing how you work, and you would want to change them anyway.

Fix patterns, never instances. The strong temptation, halfway through week one, is to start writing per-ticket exceptions. Resist it. Every instance-level fix is a permanent maintenance cost and a piece of hidden state that the next person will not understand. If forty percent of your disagreements come from one branch naming inconsistency, the correct intervention is one conversation about branch naming.

What good looks like at the end

You are ready to leave shadow mode when you can answer these, from your log, without hedging:

• What proportion of detected events would we have been happy for the system to action?
• What are the top three categories of disagreement, and have we addressed the ones that are ours to address?
• At what confidence level do errors start to cluster, and is that where our threshold is set?
• Which specific actions are we enabling first, and what is the worst thing that happens if each is wrong?
• Who reviews the queue of low-confidence items, and how often?

If you cannot answer the last one, you are not ready, regardless of your accuracy number. An automation with a human fallback queue that nobody owns is an automation with a silent backlog.

The variant nobody talks about: shadow mode as an audit

There is a second use for this that has nothing to do with rollout, and I think it is underrated.

Run shadow mode on a team with no intention of ever enabling writes, and read the log as a diagnostic report on the team’s process. It will tell you, mechanically and without politics, how much work is happening outside the record, how long the record lags reality, how many tickets describe more than one change, and how much of the backlog has criteria that cannot be evaluated.

That is a process audit that would take a consultant three weeks of interviews to produce worse. And because it is derived from source control rather than from asking people, nobody can dispute it by remembering things differently.

I have seen teams get more value from two weeks of shadow logs than from the subsequent automation, because the logs told them the problem was their refinement practice rather than their tooling. That is a completely legitimate outcome and you should be open to it.

Where this breaks down

Shadow mode is the advice in this series I am most confident about, which is exactly why it deserves the hardest look.

It cannot observe what it would have changed. This is the fundamental limit and it is not fixable. Shadow mode tells you what the system would have done to a world in which it was not acting. But once it acts, the world changes: developers stop updating tickets manually because the system does it, refinement changes because criteria now matter, and the shape of the input stream shifts. Your shadow accuracy is measured on a distribution that stops existing the moment you go live. It is a good estimate, not a guarantee, and the gap is largest exactly where behaviour change is largest.

Two weeks of a quiet sprint tells you about quiet sprints. If shadow mode runs over a period with no incidents, no hotfixes, no reverts and no release crunch, you have calibrated on the easy case. The failures that hurt are concentrated in the messy weeks: the emergency fix that skips the branch convention, the revert that makes a merged ticket un-done, the giant merge at the end of a release. If your shadow window happened to be calm, either extend it or go live with a lower risk appetite than your number suggests.

It has a real cost in momentum, and momentum is not free. Two to four weeks where nothing visibly improves is enough time for a sponsor to lose interest, for a reorg to land, or for the team to conclude that this is another initiative that will fizzle. I have watched shadow mode kill rollouts by boring them to death. If your organisation’s attention span is genuinely short, the right move may be a compressed one week shadow on a single repository, accepting a weaker calibration in exchange for still being alive in month two. That is a trade, not a mistake, but you should make it deliberately rather than by drifting.

And on a team with very few events, the log is not statistically meaningful. If the pilot team merges eight pull requests in two weeks, a hundred percent agreement rate across eight events is not evidence of anything. Small teams need longer windows or a lower-stakes go-live, not a shorter one. There is a real floor below which shadow mode is theatre, and roughly speaking it is a few dozen detected events.

The takeaway

Turn everything on, in detection, and turn every write off. Run it for a full cycle of your team’s real rhythm. Review the log weekly with someone who knows what the branches mean. Sort the disagreements into categories and fix the categories that are yours. Then set your threshold where your own data says errors begin.

The fortnight is not a delay. It is the difference between going live with a number and going live with a hope, and between being wrong in private and being wrong in front of the team whose trust you need.

The next piece takes on the other half of the same decision: which team to run this on, and why the instinct to pick the most enthusiastic one is usually wrong.