Disaster Recovery

What is it? #

Disaster recovery is the plan for restoring service after something larger than an ordinary incident: a destroyed server, a deleted database, a compromised account, a region outage.

Backups are one input. The plan also covers order of operations, access, DNS, secrets, communication and who does what.

The test of a plan is whether someone other than its author can follow it, under pressure, at an inconvenient hour, without access to the system that is down.

Rehearsal is what turns a document into a capability. Plans that have never been executed contain assumptions that fail exactly when they matter.

Think of it like this #

A fire drill. Everyone knows there is an exit; the drill reveals that the door sticks, the alarm is inaudible in one room, and nobody knows who checks the upstairs office.

Those details only surface when you walk through it.

Simple example #

A production server is destroyed. The runbook lists the steps in order, the credentials live in a password manager outside the affected system, and the last rehearsal established that recovery takes about ninety minutes.

Code #

Scenarios worth planning for, by likelihood

accidental deletion      most likely: a DROP, a bad migration, rm -rf
server or disk failure    likely: hardware, or a destroyed instance
account compromise        credentials leaked, resources deleted by an attacker
ransomware                data encrypted, including reachable backups
region outage             the whole provider region unavailable
provider account loss     billing failure or suspension

Plan for the first three properly. The last three matter for systems
where extended downtime is unacceptable.

The runbook — stored OUTSIDE the affected system

1. Assess
   what is broken, since when, what is the user impact?
   is this recoverable in place, or is a rebuild required?

2. Communicate
   status page updated, stakeholders informed, one person coordinating

3. Access
   credentials from the password manager (not from the broken server)
   provider console access confirmed

4. Rebuild infrastructure
   new server from the setup script or infrastructure code
   firewall, users, runtime, reverse proxy

5. Restore data
   latest verified database backup
   uploaded files from object storage
   configuration and secrets from the manager

6. Verify
   health endpoint, a real user journey, data completeness
   check the most recent records match expectations

7. Switch traffic
   DNS or load balancer, with TTL already low if planned

8. Afterwards
   write up what happened, what was slow, what was missing

What people discover during a real recovery

- the only person who knows the process is unavailable
- credentials were stored on the server that is gone
- the backup restores but the uploads were never backed up
- DNS TTL is 86400, so the switch takes a day
- the setup was configured by hand over two years and is undocumented
- the restore takes four hours, not the twenty minutes assumed
- nobody knows who decides to declare a disaster

Every one of these is found by a rehearsal, at no cost.

# A rehearsal, quarterly, timed
# 1. Provision a fresh environment from code
terraform apply -var environment=dr-test
# 2. Restore the most recent production backup into it
./scripts/restore.sh --from s3://backups/latest --target dr-test
# 3. Run a real user journey against it
npx playwright test tests/smoke --config dr-test
# 4. Record the elapsed time and everything that needed improvisation
# 5. Destroy it
terraform destroy -var environment=dr-test

How it works #

The runbook is ordered because order matters under pressure. Assessing before acting prevents a hurried rebuild of something that could have been fixed in place.

Communication is second rather than last. People asking for updates consume the attention of the people fixing the problem, so a single coordinator and a status page reduce the load.

Storing credentials outside the affected system is a detail that fails frequently. Keys on the destroyed server, or documentation on an internal wiki hosted on it, are both common.

Rebuilding from a setup script or infrastructure code is what makes recovery fast. A server configured by hand over two years cannot be reproduced under pressure.

Verification means checking data completeness as well as service health. A restore from a backup two days old is a successful restore and an unsuccessful recovery if nobody notices the missing data.

DNS TTL affects the switch directly. A long TTL means hours before users reach the new address, which is why keeping production TTLs moderate is worth doing in advance.

The rehearsal produces the number that matters. A measured ninety minutes is a capability; an estimated twenty minutes is a hope.

Real-world use #

Most disasters are self-inflicted. An accidental deletion, a migration that dropped a column, or a script run against the wrong environment are far more common than hardware failure.

That shapes the plan: point-in-time recovery and short RPO matter more for accidental deletion than multi-region infrastructure does.

Account compromise is the scenario people plan for least and which removes the most options, because the attacker may have the same access as the recovery process. Separate backup credentials in a separate account are the defence.

Rehearsals consistently find missing pieces. Teams that practise recover in a fraction of the time, mostly because they are not discovering problems while users wait.

The write-up afterwards is what improves the plan. Recording what was slow and what was missing, then fixing those specific things, is how the next recovery is faster.

Common mistakes #

• A plan that has never been rehearsed, so its assumptions are untested.
• Credentials or documentation stored on the system being recovered.
• Backing up the database but not uploads, configuration or secrets.
• A long DNS TTL, delaying the traffic switch by hours.
• No named coordinator, so several people act at once without knowing.

Practice #

Write a disaster recovery runbook for your system, stored outside it, covering assessment, access, rebuild, restore, verification and traffic switch. Then rehearse it into a scratch environment, time it, and list every step that required improvisation.