StackOne
Every migration postmortem we have ever written has the same shape. Someone describes the cutover step by step, and it sounds fine, reasonable, even careful. Then someone asks how many times that exact sequence ran before it touched production, and the room goes quiet. The cutover was not the problem. The fact that it had never happened before was.
Treat the runbook as a hypothesis, not a plan
A migration plan written on a whiteboard is a hypothesis about how your system behaves under a specific sequence of operations. It stays a hypothesis until you have run it against data and traffic that resemble production closely enough to surface the failure modes production would surface. Most teams skip straight from whiteboard to maintenance window, which means the first real test of the plan is the one with customers watching.
We run every non-trivial migration against a shadow environment seeded from a scrubbed production snapshot, at production scale, at least three times before it goes anywhere near a live system. The first run finds the obvious breaks: a missing index, a lock that holds longer than expected. The second run finds the timing-dependent ones. The third run is usually boring, which is exactly the point: boring is what you are rehearsing for.
Reversibility is a feature you design in, not a fallback you hope for
The single highest-leverage decision in any migration is whether every step can be undone independently of the steps around it. Dual-write phases, expand-and-contract schema changes, feature flags that let you route a percentage of traffic to the new path: none of this is exotic, but it only works if it is planned before the first line of migration code is written, not bolted on after something breaks at 2am.
- Expand the schema before you touch a single write path
- Dual-write and backfill before you dual-read
- Cut reads over gradually, behind a flag you can revert in seconds
- Only remove the old path once the new one has carried full traffic for a full business cycle
That last step is the one teams cut for time, and it is the one that turns a clean migration into an incident three weeks later, when the old path would have caught something the new one didn't.
The rehearsal is the deliverable
Clients often ask us for a migration; what we actually deliver is a rehearsed, reversible sequence of small changes, most of which are individually forgettable. The cutover itself, when it happens, tends to be the least interesting hour of the whole engagement, which is precisely why nobody notices it happened at all.