DevOpsIntermediate 13 min Lesson 7 of 15

Backup Strategy

Decide what to back up, how often, where to keep it, and how to prove it works before you need it.

DevOps · Lesson 7 of 15
0/15 done(0%)

What is it? #

A backup strategy is a set of decisions: what is backed up, how often, where copies live, how long they are kept, and how you know they work.

Most teams have backups of the database and nothing else. Uploaded files, configuration and secrets are equally necessary for a restore and are frequently missing.

The two numbers that frame everything are RPO — how much data you can afford to lose — and RTO — how long recovery may take. Both should be written down.

Verification is what separates a strategy from a hope. A backup that has never been restored is an untested assumption.

Think of it like this #

Insurance with an excess you have never checked and a claims process you have never used.

You discover both details at the worst possible moment, which is exactly when you least want a surprise.

Simple example #

A ransomware incident encrypts the server and the attached backup share. The offsite, versioned copy in object storage is untouched, and the documented restore takes ninety minutes because it was practised last quarter.

Code #

TEXT
Scope: what actually needs backing up

database          essential — usually the only irreplaceable part
uploaded files    essential — user data, often forgotten
configuration     essential — .env files, service definitions, Nginx config
secrets           essential, encrypted — a restore without them is useless
TLS certificates  convenient; reissuable if not
application code  no — it is in git
system packages   no — reinstallable from a setup script
logs              only if required for compliance
TEXT
The 3-2-1 rule, and why each part matters

3 copies    live data plus two backups
2 media     different storage types or locations
1 offsite   a different provider or physical location

Offsite is the part that matters most. A failed disk, a deleted cloud
account, a ransomware run or a region outage can take the server and
any backup stored beside it at the same time.
TEXT
Frequency, set from RPO

RPO 24 hours    nightly full backups
RPO 1 hour      hourly incrementals plus a nightly full
RPO minutes     continuous archiving (WAL shipping) plus periodic base backups

Retention:
  daily      14-30 days   most restores are recent
  weekly     3 months     for problems discovered late
  monthly    1 year       compliance, and for slow-moving corruption
BASH
# Verification: the part that makes it real
# 1. Does the file exist and is it the expected size?
# 2. Is it readable?
pg_restore --list backup.dump > /dev/null
# 3. Does it restore?
pg_restore --dbname=verify_db backup.dump
# 4. Is the data current and complete?
psql verify_db -c "SELECT count(*), max(created_at) FROM orders;"
# 5. How long did the whole thing take? Record it — that is your real RTO.
TEXT
Protecting backups from the incident itself

immutable storage      object lock or versioning: cannot be deleted or
                       overwritten for a retention period
separate credentials   the backup destination should not be writable with
                       the same key the application uses
separate account       ransomware and a compromised account both spread
                       through shared access
encrypted at rest      backups contain everything sensitive you have

A backup an attacker can delete is not a backup.

How it works #

Scope is where strategies usually fail. A perfect database backup is not a recovery if the uploaded files are gone and nobody has the environment configuration.

The 3-2-1 rule exists because single-location failures are the common case. Offsite specifically protects against the failure that takes both the server and anything stored near it.

Frequency follows directly from RPO. Nightly backups mean up to 24 hours of loss, which is acceptable for some systems and unacceptable for others — the decision belongs to the business, not to the defaults.

Retention tiers exist because problems are discovered at different speeds. A deletion noticed in an hour needs yesterday's backup; slow data corruption noticed in March may need January's.

The verification steps are ordered from cheap to thorough. Checking that a file exists is nearly free and can run daily; a full restore is expensive and belongs on a quarterly schedule.

Recording the restore duration gives you the real RTO. Assumed recovery times are consistently optimistic.

Immutability is the modern requirement. Ransomware deliberately targets backups, and object lock or versioning means a compromised credential cannot delete history. Using separate credentials for the backup destination limits that further.

Real-world use #

The most common backup failure is not absence but incompleteness: the database was covered and user uploads were not.

Ransomware changed the requirements. Backups on a mounted network share get encrypted with everything else, which is why object storage with versioning has become standard.

Cloud provider snapshots are convenient and insufficient on their own, because they live in the same account. An account compromise or a billing suspension can remove them.

Restore rehearsals reveal missing pieces reliably: nobody has the encryption key, the documentation refers to a server that no longer exists, or the restore takes four hours when everyone assumed twenty minutes.

Testing quarterly, timed and documented, converts a stack of files into a measured recovery capability.

Common mistakes #

  • Backing up the database but not uploads, configuration or secrets.
  • Keeping the only backup in the same account or on the same machine.
  • Backups an attacker or a mistake can delete — no immutability.
  • Never measuring the actual restore time, so RTO is a guess.
  • No alert when a backup fails or simply stops running.

Practice #

Write down your RPO and RTO, then list everything needed for a full restore and check whether each is currently backed up. Perform a restore into a scratch environment, time it, and compare the result with your stated RTO.

Quick quiz

  1. 1. What is most commonly missing from a backup strategy?

  2. 2. Why does the offsite copy matter most?

  3. 3. What determines backup frequency?

  4. 4. Why use immutable or versioned backup storage?

  5. 5. How do you determine your real RTO?

Summary

  • Back up the database, uploads, configuration and secrets.
  • Follow 3-2-1, with the offsite copy as the critical part.
  • Set frequency and retention from a written RPO.
  • Use immutable storage and separate credentials for backups.
  • Test restores on a schedule and record the actual time.