What is it? #
Rollback is returning production to the previous known-good version. In a pipeline, it should be automatic for obvious failures and a single action otherwise.
The code side is straightforward when you keep previous artefacts: deploy the previous image or switch the release symlink.
The database side constrains everything. A destructive migration removes the rollback option, which is why backwards-compatible schema changes are a prerequisite for safe deployment.
Rollback must be tested. A path that has never been exercised is not a recovery plan.
Think of it like this #
A reverse gear that you have actually used. Knowing it exists is not the same as knowing it engages smoothly under load.
Practising in an empty car park is what makes it usable in traffic.
Simple example #
A deployment whose health check fails triggers an immediate automatic rollback. A subtler problem found twenty minutes later is rolled back by running a workflow with the previous version as input.
Code #
# Automatic rollback when verification fails
- name: Verify deployment
id: verify
run: ./scripts/verify-production.sh ${{ github.sha }}
- name: Roll back
if: failure() && steps.verify.outcome == 'failure'
run: |
PREVIOUS=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>g</mi><mi>h</mi><mi>a</mi><mi>p</mi><mi>i</mi><mi>r</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>s</mi><mi mathvariant="normal">/</mi></mrow><annotation encoding="application/x-tex">(gh api repos/</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal">ha</span><span class="mord mathnormal">p</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">os</span><span class="mord">/</span></span></span></span>{{ github.repository }}/deployments \
--jq '[.[] | select(.environment=="production")][1].sha')
ssh deploy@prod "IMAGE=ghcr.io/<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>g</mi><mi>i</mi><mi>t</mi><mi>h</mi><mi>u</mi><mi>b</mi><mi mathvariant="normal">.</mi><mi>r</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>s</mi><mi>i</mi><mi>t</mi><mi>o</mi><mi>r</mi><mi>y</mi></mrow><mo>:</mo><mi>s</mi><mi>h</mi><mi>a</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">{{ github.repository }}:sha-</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord"><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">os</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.0278em;">or</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">ha</span><span class="mord">−</span></span></span></span>PREVIOUS /srv/app/deploy.sh"
./scripts/verify-production.sh "$PREVIOUS"
- name: Notify
if: always()
run: ./scripts/notify.sh "${{ job.status }}"# A manual rollback workflow — one click, no improvisation
name: Rollback production
on:
workflow_dispatch:
inputs:
version:
description: 'Commit SHA or image tag to roll back to'
required: true
jobs:
rollback:
runs-on: ubuntu-latest
environment: production
steps:
- name: Confirm the image exists
run: docker manifest inspect ghcr.io/<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>g</mi><mi>i</mi><mi>t</mi><mi>h</mi><mi>u</mi><mi>b</mi><mi mathvariant="normal">.</mi><mi>r</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>s</mi><mi>i</mi><mi>t</mi><mi>o</mi><mi>r</mi><mi>y</mi></mrow><mo>:</mo><mi>s</mi><mi>h</mi><mi>a</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">{{ github.repository }}:sha-</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord"><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">os</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.0278em;">or</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">ha</span><span class="mord">−</span></span></span></span>{{ inputs.version }}
- name: Deploy previous version
run: ssh deploy@prod "IMAGE=ghcr.io/<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>g</mi><mi>i</mi><mi>t</mi><mi>h</mi><mi>u</mi><mi>b</mi><mi mathvariant="normal">.</mi><mi>r</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>s</mi><mi>i</mi><mi>t</mi><mi>o</mi><mi>r</mi><mi>y</mi></mrow><mo>:</mo><mi>s</mi><mi>h</mi><mi>a</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">{{ github.repository }}:sha-</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord"><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">os</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.0278em;">or</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">ha</span><span class="mord">−</span></span></span></span>{{ inputs.version }} /srv/app/deploy.sh"
- name: Verify
run: ./scripts/verify-production.sh "${{ inputs.version }}"What makes rollback possible
previous artefacts kept do not prune the last N images
immutable tags the previous version is addressable
backwards-compatible
migrations the old code still works against the new schema
externalised config no rebuild needed to change environment settings
feature flags risky behaviour can be disabled without deployingRollback triggers
automatic health check fails after deployment
error rate above a threshold within N minutes
the deployed version does not match what was deployed
manual a subtle problem discovered later
a business decision to revert a feature
Automatic triggers must be reliable. A rollback fired by a flaky check
is its own incident.Roll back or fix forward?
roll back the failure is immediate and clear
the previous version is known good
no irreversible migration has run
fix forward the migration cannot be reversed
the previous version has its own serious problem
the fix is small, understood and already tested
Default to rolling back. Restore service first, investigate afterwards.How it works #
Automatic rollback is attached to the verification step. If the new version does not become healthy, the pipeline redeploys the previous image and verifies that instead.
Finding the previous version means querying deployment history rather than assuming. Hardcoding "the previous tag" breaks when two deployments happen in quick succession.
The manual workflow takes a version as input and performs the same deployment path. Using the normal deployment script rather than a special rollback procedure means the path is exercised constantly.
Checking that the image exists before deploying prevents the worst case: a rollback that fails because the artefact was pruned, leaving production broken with no quick path back.
Feature flags reduce how often rollback is needed at all. A feature deployed disabled and enabled for a subset can be turned off in seconds without touching the deployment.
Notification on every outcome matters. A silent automatic rollback means production is running an older version and nobody knows why the new feature is missing.
The trigger reliability point is worth emphasising. An automatic rollback fired by a flaky health check causes an unnecessary deployment cycle, which is why verification must be as trustworthy as the tests.
Real-world use #
Teams that can roll back in under a minute deploy far more often, because the cost of being wrong is small.
The constraint is almost always the database. Once a migration has dropped a column, rolling back the code means the old version queries a column that no longer exists.
Artefact retention policy matters more than it seems. Pruning aggressively saves storage and removes rollback targets; keeping the last twenty images is cheap insurance.
Canary and blue-green deployments make rollback nearly instant by keeping the previous version running and switching traffic back, which the next lesson covers.
Practising rollback during a calm period — deploying a trivial change and reverting it — is what makes it reliable when it is needed.
Common mistakes #
- Destructive migrations that make rollback impossible.
- Pruning old images aggressively, removing the rollback target.
- A rollback path that is never tested until an incident.
- Automatic rollback triggered by an unreliable check.
- Rolling back silently, so nobody knows production changed.
Practice #
Add automatic rollback on verification failure to a deployment pipeline, plus a manual rollback workflow taking a version as input. Deploy a deliberately broken version and confirm the automatic rollback restores service and notifies the team.