What is it? #
Rollback is returning to the previous working version quickly when a deployment goes wrong.
With release directories and a symlink, the code side is trivial: point current at the previous release and restart. Seconds, not minutes.
The database is the hard part. Code can be reverted; a migration that dropped a column cannot be undone by switching a symlink.
The technique that makes rollback safe is backwards-compatible migrations: never make a schema change that the previous version of the code cannot tolerate.
Think of it like this #
Keeping the old stock in the back room for a week after a product change.
Putting it back on the shelf takes minutes. What you cannot undo is having thrown it away — which is what a destructive migration does.
Simple example #
A deployment fails its health check. The rollback script switches the symlink back, restarts the service, verifies health and reports what happened — all within thirty seconds.
Code #
#!/usr/bin/env bash
# /srv/app/rollback.sh — keep this ready before you need it
set -euo pipefail
APP=/srv/app
CURRENT=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>r</mi><mi>e</mi><mi>a</mi><mi>d</mi><mi>l</mi><mi>i</mi><mi>n</mi><mi>k</mi><mo>−</mo><mi>f</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">(readlink -f "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">a</span><span class="mord mathnormal">d</span><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="mord mathnormal" style="margin-right:0.0315em;">ink</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.1076em;">f</span><span class="mord">"</span></span></span></span>APP/current")
PREVIOUS=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>l</mi><mi>s</mi><mo>−</mo><mn>1</mn><mi>d</mi><mi>t</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">(ls -1dt "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.6944em;"></span><span class="mord">1</span><span class="mord mathnormal">d</span><span class="mord mathnormal">t</span><span class="mord">"</span></span></span></span>APP"/releases/* | grep -v "^<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>U</mi><mi>R</mi><mi>R</mi><mi>E</mi><mi>N</mi><mi>T</mi></mrow><annotation encoding="application/x-tex">CURRENT</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.0715em;">C</span><span class="mord mathnormal" style="margin-right:0.109em;">U</span><span class="mord mathnormal" style="margin-right:0.0077em;">R</span><span class="mord mathnormal" style="margin-right:0.0077em;">R</span><span class="mord mathnormal" style="margin-right:0.0576em;">E</span><span class="mord mathnormal" style="margin-right:0.109em;">N</span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span></span></span></span>" | head -1)
[ -z "$PREVIOUS" ] && { echo "no previous release available"; exit 1; }
echo "rolling back: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>b</mi><mi>a</mi><mi>s</mi><mi>e</mi><mi>n</mi><mi>a</mi><mi>m</mi><mi>e</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">(basename "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">ba</span><span class="mord mathnormal">se</span><span class="mord mathnormal">nam</span><span class="mord mathnormal">e</span><span class="mord">"</span></span></span></span>CURRENT") -> <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>b</mi><mi>a</mi><mi>s</mi><mi>e</mi><mi>n</mi><mi>a</mi><mi>m</mi><mi>e</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">(basename "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">ba</span><span class="mord mathnormal">se</span><span class="mord mathnormal">nam</span><span class="mord mathnormal">e</span><span class="mord">"</span></span></span></span>PREVIOUS")"
ln -sfn "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mi>R</mi><mi>E</mi><mi>V</mi><mi>I</mi><mi>O</mi><mi>U</mi><mi>S</mi><mi mathvariant="normal">"</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">PREVIOUS" "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.6944em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">P</span><span class="mord mathnormal" style="margin-right:0.0077em;">R</span><span class="mord mathnormal" style="margin-right:0.0576em;">E</span><span class="mord mathnormal" style="margin-right:0.2222em;">V</span><span class="mord mathnormal" style="margin-right:0.0785em;">I</span><span class="mord mathnormal" style="margin-right:0.0278em;">O</span><span class="mord mathnormal" style="margin-right:0.109em;">U</span><span class="mord mathnormal" style="margin-right:0.0576em;">S</span><span class="mord">""</span></span></span></span>APP/current.new"
mv -T "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>A</mi><mi>P</mi><mi>P</mi><mi mathvariant="normal">/</mi><mi>c</mi><mi>u</mi><mi>r</mi><mi>r</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>n</mi><mi>e</mi><mi>w</mi><mi mathvariant="normal">"</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">APP/current.new" "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.1389em;">P</span><span class="mord mathnormal" style="margin-right:0.1389em;">P</span><span class="mord">/</span><span class="mord mathnormal">c</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal">n</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.0269em;">w</span><span class="mord">""</span></span></span></span>APP/current"
sudo systemctl restart app
for i in {1..15}; do
curl -fsS http://127.0.0.1:8000/health > /dev/null && { echo "healthy"; exit 0; }
sleep 2
done
echo "ROLLBACK FAILED HEALTH CHECK — escalate"
exit 1Backwards-compatible migrations: expand and contract
Adding a column
deploy 1: add the column, nullable, with a default ← old code ignores it
deploy 2: code that writes and reads it
deploy 3: make it NOT NULL once every row is populated
Removing a column
deploy 1: code stops reading and writing it ← rollback-safe
deploy 2: (after a safe interval) drop the column
Renaming a column
never rename in place. Add the new one, write to both, backfill,
switch reads, stop writing the old one, then drop it.
The rule: at every step, the previous release must still work.Roll back or fix forward?
Roll back when
the failure is immediate and obvious
the previous version is known good
no destructive migration has run
the fix is not understood yet
Fix forward when
a migration cannot be reversed
the old version has its own serious problem
the fix is small, understood and tested
Default to rolling back. Debugging a production incident with users
affected is slower and riskier than restoring a known-good state.# Verify rollback works BEFORE you need it
./deploy.sh # deploy a trivial change
./rollback.sh # roll it back
curl -s https://app.example.com/version # confirm the old version is servingHow it works #
readlink -f resolves the current symlink to its real target, and listing releases by modification time finds the most recent one that is not current.
The switch uses the same atomic symlink replacement as deployment, so there is never a moment with no valid target.
The health check after the rollback matters as much as after a deployment. A rollback that also fails tells you the problem is not in the new code — perhaps the database or a dependency — which changes what you do next.
The expand-and-contract pattern is the core idea for migrations. Every schema change is split so that at no point does the running code disagree with the schema. Adding a nullable column is invisible to old code; dropping a column that old code still reads breaks it immediately.
Renaming is the case people get wrong. An in-place rename breaks both directions at once, which is why the six-step version exists.
The roll back versus fix forward guidance is worth internalising before an incident. Under pressure, the instinct is often to debug, and restoring a known-good state first is almost always the better order.
Testing rollback during a calm afternoon is what makes it usable during a bad one.
Real-world use #
Teams that can roll back in thirty seconds deploy far more often, because the cost of a bad deployment is small.
Migration discipline is what makes it real. A single destructive migration removes the rollback option for that deployment and everything after it until the data is restored.
Feature flags reduce the need for rollback entirely: deploy the code disabled, enable it for a small group, and turn it off instantly if something is wrong. That is a configuration change rather than a deployment.
With a load balancer and two servers, a canary deployment — new version on one server, observed, then rolled out — catches problems before they affect everyone.
The rollback script should be written and tested when the deployment script is, not improvised during an incident.
Common mistakes #
- Destructive migrations that make rollback impossible.
- Renaming columns in place instead of expand and contract.
- No rollback script, so recovery is improvised under pressure.
- Deleting old releases too aggressively, leaving nothing to roll back to.
- Debugging in production instead of restoring a known-good state first.
Practice #
Write a rollback script for your deployment structure and test it by deploying a change and reverting it. Then plan the expand-and-contract sequence for renaming a database column, listing every deployment step and what the previous version does at each point.