Skip to main content

Disaster Recovery

Every failure mode covered so far in this module — a dead node, a failing dependency, a partitioned cluster — assumes the rest of the system is still there to react. Disaster recovery is what happens when the failure is bigger than that: an entire datacenter loses power, an entire cloud region goes offline, or data is corrupted or deleted at a scale no single-node failover can fix. It's the plan for "what if the thing we usually failover to is also gone."

Redundant servers don't help against this​

A Single Point of Failure analysis usually stops at "redundant servers, redundant load balancers, redundant database replicas" — but if every one of those redundant copies lives in the same building, a single power outage, fire, or network cut takes out all of them simultaneously, and none of that redundancy helps. This is exactly the gap that lesson names as the motivation for multi-region disaster recovery: real resilience against a disaster-scale event means the redundant copies have to be geographically separated, not just logically separated within one location.

The two numbers that define a DR plan​

Every disaster recovery strategy is ultimately a tradeoff between cost and two measurable targets, and naming them precisely is what separates a real DR plan from a vague promise of "we have backups":

  • RPO (Recovery Point Objective) — how much data can be lost, measured in time. An RPO of 5 minutes means that after a disaster, up to 5 minutes of the most recent writes might be gone.
  • RTO (Recovery Time Objective) — how long the system can be down before it's back up. An RTO of 1 hour means the system must be fully operational again within an hour of the disaster starting.

Both numbers directly drive cost: a smaller RPO needs more frequent (or fully synchronous) replication to the recovery site, and a smaller RTO needs infrastructure standing by ready to take over immediately rather than being provisioned from scratch after the fact.

The standard strategies, by cost and speed​

System Design Lab
  • Backup and restore — periodic backups stored in a separate region; recovery means provisioning fresh infrastructure and restoring from backup. Cheapest, but slowest (RTO measured in hours) and loses anything written since the last backup (RPO bounded by backup frequency).
  • Pilot light — a minimal version of the core infrastructure (often just the database, kept continuously replicated) sits idle in the recovery region; everything else is provisioned and scaled up only when disaster strikes. Faster recovery than a cold backup, at moderate ongoing cost.
  • Warm standby — a scaled-down but fully functional copy of the whole system runs continuously in the recovery region, ready to take over and scale up on demand. Meaningfully faster RTO, at meaningfully higher ongoing cost (you're paying to run infrastructure that's usually idle).
  • Multi-region active-active — both regions serve live production traffic simultaneously, so "failover" is really just DNS or load balancing removing the failed region from rotation — the exact layered pattern (DNS picks a region, a regional load balancer picks a server) already named as disaster recovery's standard shape. Fastest possible RTO (near-zero) and best possible RPO, at the highest cost and the most operational complexity, since both regions' data has to actually stay in sync.

Why this is expensive, and why that's the actual point​

Every strategy above trades cost for how close RPO and RTO get to zero — there's no free version of "instant recovery with no data loss," because achieving it means paying to keep a second, geographically separate copy of the entire system continuously warm and continuously synchronized. The real skill being tested in an interview isn't naming multi-region active-active as "the best" option; it's matching the strategy to what the business actually needs. A backend for an internal analytics dashboard can likely tolerate hours of downtime and some data loss; a payments system almost certainly cannot — and the DR strategy, and its cost, should follow directly from that answer.

Why this matters in an interview​

Naming specific RPO/RTO targets — even rough ones — turns "we'll have disaster recovery" into a concrete, defensible requirement instead of a box to check. Walking through why a given strategy fits those targets (and explicitly not over-engineering toward active-active for a workload that doesn't need it) demonstrates the same cost-aware judgment this course has emphasized throughout: more redundancy is a tradeoff, not a free win.

Warm standby vs. multi-region active-active: pros and cons​

Warm standby

  • Meaningfully lower ongoing cost than running full production capacity in two regions
  • Simpler operational model — only one region serves live traffic at a time
  • Still a large RTO improvement over backup-and-restore or pilot light

Active-active

  • Highest ongoing infrastructure cost — both regions run at full production capacity
  • Requires solving multi-region data synchronization and conflict resolution continuously
  • Substantially more operational complexity to run and reason about correctly

Further Reading​

Saved locally in your browser — visible in the sidebar as you go.