Skip to main content

Failover

Failover is the process of automatically switching to a redundant or standby component when the active one fails. It's the mechanism that turns "we have a backup" into "the backup is actually helping" — redundancy without failover is just a spare part sitting on a shelf, useless until someone notices the failure and manually swaps it in.

Failover is what actually connects redundancy to availability and fault tolerance: redundancy provides the spare capacity, and failover is the automated process that uses it in time to matter.

System Design Lab

The three parts of any failover system​

Every failover setup, no matter how it's implemented, needs to solve three problems:

  1. Detection — noticing that the active component has actually failed. This is usually done with health checks or heartbeats: the standby (or a separate monitor) pings the active component on a fixed interval, and if it misses enough consecutive responses, it's declared dead.
  2. Decision — deciding which standby should take over, and confirming there isn't already another node that thinks it's the active one (more on why this matters below).
  3. Redirection — actually routing traffic to the new active component. This might mean a load balancer marking the old target unhealthy and removing it from rotation, a DNS record changing, or a database replica being promoted to primary and clients reconnecting to the new address.

Active-passive vs. active-active​

  • Active-passive failover: one component handles all traffic; one or more standbys sit idle (or replicate data in the background) and only take over on failure. Simpler to reason about, but the standby's capacity is otherwise wasted, and there's often a brief gap (failover time) where nothing is serving traffic while the switch happens.
  • Active-active failover: multiple components handle traffic simultaneously, so if one fails, the others just absorb its share of load — there's no separate "become active" step, just a reduction in capacity until the failed node is replaced. This gives faster recovery (nothing has to be "promoted") but requires the system to already be designed for multiple concurrent writers/handlers, which is a harder property to guarantee, especially for stateful components like databases.

The classic failure mode: split-brain​

The most important thing to get right in failover design is avoiding split-brain: a situation where a network partition makes the standby believe the primary is dead (so it promotes itself), while the original primary is actually still alive and still thinks it's in charge. Now you have two nodes both accepting writes as if they're the sole primary — and reconciling the diverged data afterward can be genuinely difficult or impossible without data loss.

The standard defense is quorum-based decision-making: instead of a standby unilaterally deciding "I haven't heard from the primary, so I'll take over," a majority of nodes in the cluster must agree that the primary is actually down before a new one is promoted. This is closely related to Consensus Algorithms like Raft and Paxos, which exist specifically to make this kind of "who's in charge" decision safely in the presence of partitions and node failures.

Where failover shows up in interview designs​

  • Load balancers performing health checks and removing unhealthy backend servers from rotation — the simplest and most common form of failover you'll design in an interview.
  • Database replicas being promoted to primary when the primary fails (most managed database services, like RDS Multi-AZ, automate exactly this).
  • DNS failover, where a DNS provider monitors an endpoint and switches the returned IP address if it becomes unreachable — slower (subject to DNS caching/TTLs) but useful for failing over across entire regions or data centers.

When you propose failover in a design, it's worth naming the detection mechanism and the expected failover time (how long until traffic is fully redirected) — "we have a standby" is a much weaker answer than "a health check fails after 3 missed pings at 5-second intervals, so failover takes roughly 15–20 seconds."

Active-active over active-passive: pros and cons​

Pros

  • No idle standby capacity — every node is already doing useful work
  • Faster recovery: no "promote the standby" step, just reduced capacity
  • Load is already spread, so one failure is a smaller relative hit

Cons

  • Requires a design that safely tolerates multiple concurrent writers
  • Harder to reason about correctness than a single clear "active" node
  • Not all stateful systems (especially databases) support this safely

Further Reading​

Saved locally in your browser — visible in the sidebar as you go.