The rollback plan you should have before you start

Ask a team midway through one of these rollouts what they would do if it started behaving badly, and you get a shrug and a reasonable-sounding sentence: we would turn it off.

Turn what off, exactly? All of it, or the specific action misbehaving? What happens to the four hundred tickets it has already updated? Does the board revert to what it said before, or freeze where it is? Who has the permission to do it at eleven at night without waking anyone? And after you turn it off, how does anyone know what the real status of anything is, given that nobody has updated a ticket by hand in six weeks?

None of those are hard questions. They are just questions nobody has answered, and the time to answer them is not while the thing is going wrong.

A rollback plan for this is a single page. It takes forty-five minutes to write. It is the cheapest confidence you will buy in the whole transition, and its real value is not the disaster you avoid but the conversation it forces you to have about what you are actually depending on.

Why this is not a normal rollback

Rolling back a deployment is well understood: you have an artefact, you have a previous artefact, you swap them. The rollback of a reconciliation layer is different in three ways that make the usual instincts unhelpful.

The side effects live in someone else’s system. When you disable the automation, the tickets it moved stay moved. The comments it posted stay posted. Your tracker is now in a state produced partly by software you have just switched off, and nothing about disabling it reverses that.

There is no single state to return to. A deployment rollback returns to a known previous version. Here, the previous state is a board that was four days out of date, and getting back to it is neither possible nor desirable. What you actually want is not the old state but a correct state, which is a different target.

Human process atrophies while automation runs. This is the one people miss. After six weeks of automatic transitions, nobody on the team has moved a ticket by hand. The habit is gone, and the knowledge of who was supposed to do it is fuzzy. Turning the automation off does not restore the manual process, it just removes the automated one and leaves a vacuum.

That third point is the strongest argument for having the plan in writing before you start, because it is the one that bites hardest and it is invisible until it happens.

After six weeks of automatic transitions, nobody on the team has moved a ticket by hand. The habit is gone.

So a rollback does not return you to where you started. It returns you to a team that has forgotten how to do the thing you are asking them to resume, which is invisible until the moment it matters.

The four levels of rollback

The single most useful thing in the plan is a graduated ladder, because “turn it off” is almost never the right first response and having only that option means you will either overreact or delay.

Level one: raise the threshold. The fastest and least disruptive intervention. Push the auto-apply confidence threshold up, and more decisions route to a human instead of executing. The automation keeps running, keeps detecting, keeps proposing, and simply acts less often. This handles the most common real scenario, which is not “it is broken” but “it is being wrong more than I am comfortable with in one specific area.” Takes seconds and nothing is lost.

Level two: disable a specific action. Turn off status transitions but keep linking and evidence. Turn off the merge gate but keep everything else. This is the right response to a categorical problem, and it is the level most incidents should end at. It preserves the value you have while removing the specific thing misbehaving.

Level three: back to shadow. Detection continues, all writes stop. Nothing changes in the tracker, but the system keeps logging what it would have done. This is materially better than a full stop for one reason: you keep generating diagnostic data about the very problem you are investigating. If you turn everything off, you have also turned off your ability to work out what went wrong.

Level four: disconnect. Full stop, integration removed. This should be genuinely rare, and it is mostly reserved for security or access concerns rather than correctness ones. Correctness problems are level one to three problems.

Write those four levels down with, for each: who can do it, how long it takes, what stops happening, and what state you are left in. That is most of your page.

Level one: raise the thresholdFastest and least disruptive. More decisions route to a human instead of executing. Nothing is lost, only slowed.

Level two: disable one actionTransitions off, linking and evidence on. The right response to a categorical problem rather than a general one.

Level three: back to shadowDetection continues, all writes stop. Materially better than disconnecting, because you keep learning.

Level four: disconnectGenuinely rare, and mostly for security or access concerns rather than correctness ones.

The undo question, answered properly

The thing everyone wants to know is whether the writes can be reversed, and the answer has a shape worth understanding rather than a simple yes or no.

Reversibility depends on whether the prior value was captured at the moment of the write. If it was, then reverting a transition is mechanical: the system knows the ticket said in progress before it said done, so it can put it back. This is a property you should verify before you enable anything, and it is a reasonable thing to ask a vendor to demonstrate rather than assert. Show me a write, show me the record of what it was before, show me the revert.

What is not reversible, in any system, is anything that left the boundary. A notification that fired. An email digest that went out. A webhook that triggered a downstream job. A stakeholder who read a report on Tuesday. This is the concrete reason the previous article in this series put “anything that messages people outside the team” in the do-not-automate tier: not because the inference is worse, but because the undo does not exist.

So your plan needs two lists: what can be reverted and what has already left the building. The second list is short and it is the one that matters.

Already left the building

• A notification that fired
• An email digest that went out
• A webhook that triggered a downstream job
• A stakeholder who read it

Can be reverted

• Transitions, if the prior value was captured
• Comments, removable with the record kept
• Links, one click each
• Anything still inside your own systems

Bulk revert, and its trap

At some point you may want to undo not one write but a batch: everything from a bad afternoon, or everything touching a particular project.

Two rules.

Never bulk-revert blind. A batch revert of six hours of transitions will also revert the correct ones, and correct transitions are the large majority even in a bad batch. You will have turned a hundred-item problem with twelve errors into a hundred-item problem with eighty-eight errors, in the opposite direction, and now nobody trusts the board at all.

Filter first, then confirm, then revert. Select by time window and action type, review the list, exclude the ones that were right. This is tedious and it is correct. If your review shows that most of the batch was fine, the right move is usually to fix the twelve individually and leave the rest.

There is a related trap worth naming: reverting a status and then leaving it. If a ticket goes back to in progress and the underlying work really did merge, you have created a new inaccuracy of the opposite kind. Every revert needs a follow-up owner, or you have traded a wrong-forward board for a wrong-backward one.

Restarting the human process

This is the part of the plan people never write and always need.

If you go to level three or four, someone has to maintain the record by hand again, and after six weeks nobody remembers that it was ever their job. Your page needs three lines about this:

• Who updates tickets manually during a rollback. A named role, not “the team.”
• How the team is told. One message, one channel, with the specific sentence “status is no longer updating automatically, please move your own tickets.”
• How long before the board is untrustworthy. In most teams this is under a week. Knowing the number tells you how urgent the restart is.

If you have been running long enough that the manual process is genuinely forgotten, the honest recovery is a one-hour manual reconciliation session where the team walks the board and fixes it together. Ugly, effective, and much better than pretending the board is fine.

Rehearse it once

Write the plan, then actually execute a level one and a level two in week two of live writes, deliberately, on a normal day when nothing is wrong.

Raise the threshold, watch what happens to the human queue, and lower it again. Disable one action, confirm the others keep running, re-enable it. It takes twenty minutes.

The value is not that you learn the buttons. It is that the team watches a controlled reversal happen without drama, which does more for their confidence than any assurance you can give. It also reliably surfaces the thing your plan got wrong, which in my experience is usually the permission question: the person named in the plan turns out not to have the access, and nobody would have discovered that until the night it mattered.

What the page contains

To be concrete, the whole artefact is:

• The four levels, with who, how long, and what stops.
• The revertible and not-revertible lists.
• The named individual who can execute each level, and the confirmation that they have the access today.
• The bulk revert procedure, with the filter-review-confirm rule.
• The manual process restart: who, what message, how long before the board rots.
• The date you last rehearsed it.

One page. Written before go-live. Reviewed when you expand.

Where this breaks down

Three real limits.

The plan protects you from correctness failures far better than from trust failures. If the automation is wrong in a way that is visible and embarrassing, the damage is not the wrong ticket, it is the team’s belief that the system is unreliable. You can revert the ticket in four seconds. You cannot revert the belief, and a rollback plan does nothing about it. This is why the sequencing advice matters more than the recovery advice: the plan is insurance, not a substitute for going slowly enough that the first week is clean.

Reverting is not always the right response, and a well-rehearsed plan biases you toward using it. If the automation moved forty tickets to done and thirty-eight were correct, reverting the batch is worse than fixing two. Having a rehearsed procedure creates a pull toward executing it, because it is the action you know how to take. The discipline is to ask whether the automation was wrong or whether it was right and someone was surprised, and those feel identical in the first ten minutes.

And a rollback plan can become a substitute for judgement about what to enable. I have seen the reasoning: we can always revert, so we can afford to turn on the aggressive thing. That is exactly backwards for anything whose effects leave the system, and it is subtly wrong even for things that do not, because the cost of an error is rarely just the mechanical undo. Reversibility lowers the cost of being wrong. It does not lower it to zero, and treating it as though it does is how teams end up enabling merge gates in month one.

The takeaway

Write the page before you go live. Four levels rather than one switch, because most incidents are threshold problems rather than disconnect problems, and back-to-shadow is almost always better than off. Know what is revertible and what has already left the building. Never bulk-revert blind. Name who restarts the manual process, because six weeks of automation erases the habit.

Then rehearse a level one and a level two on an ordinary Tuesday, which will find the permissions problem you did not know you had and will do more for the team’s confidence than any promise.

The next piece is about the dial you will reach for first in almost every one of these situations: how to choose a confidence threshold, and why the number that feels safe is usually not the one your data supports.