Fault Tolerance

Fault tolerance is a system's ability to keep operating correctly even when some of its components fail. Note the phrasing carefully: it's not about preventing failures — at any meaningful scale, you cannot prevent hardware from dying, networks from dropping packets, or processes from crashing. Fault tolerance is about surviving failures that are assumed to be inevitable, so they don't turn into outages.

This is a subtly different goal from reliability (which is about a component not failing in the first place) and from failover (a specific mechanism for switching to a backup). Fault tolerance is the broader property: a fault-tolerant system might use failover, redundancy, retries, and graceful degradation together, and the test of whether it worked is simple — did a component failing actually affect the user?

Fault tolerance vs. high availability

These two get conflated often enough that it's worth being precise, because interviewers listen for this:

• High availability minimizes downtime — the system is designed to come back up fast after a failure (often via failover), but there may be a brief gap where it's not serving requests.
• Fault tolerance aims for zero perceptible impact — the system keeps running through the failure with no interruption at all, typically because redundant components were already actively sharing the work (active-active), so losing one just means the others carry a bit more load.

A useful way to say this out loud in an interview: "a highly available system recovers quickly from failure; a fault-tolerant system doesn't stop during it."

System Design Lab

Bulkheading in action: Service A's pool absorbs the failure — Service B's separate pool keeps serving normally.

Core techniques

• Redundancy — running multiple instances of a component so the loss of one doesn't remove the capability entirely (see Single Point of Failure).
• Replication — keeping multiple synchronized copies of data, so a storage node failing doesn't lose data (see Data Replication).
• Isolation / bulkheading — physically or logically partitioning resources (thread pools, connection pools, even entire services) so that a failure or resource exhaustion in one part can't consume resources needed by another. Named after ship bulkheads, which keep one flooded compartment from sinking the whole vessel.
• Timeouts and circuit breakers — preventing a slow or failing dependency from holding resources indefinitely or letting failures cascade upstream (see Circuit Breaker).
• Graceful degradation — when a non-essential dependency fails, serve a reduced but still useful response instead of failing the whole request (e.g., an e-commerce page that renders without personalized recommendations if that microservice is down, rather than showing an error for the entire page).
• Retries with backoff — for transient failures, retrying (ideally with exponential backoff and jitter, to avoid a "thundering herd" of retries making things worse) can make a fault invisible to the end user entirely, provided the operation is safe to repeat (see Idempotency).

Redundancy alone isn't fault tolerance

A common interview mistake is treating "I added 3 replicas" as equivalent to "this is fault tolerant." Redundancy is necessary but not sufficient — you also need:

• A way to detect that one replica has failed (otherwise you're silently running on reduced capacity, or worse, silently serving stale/wrong data from a half-broken node).
• A way to route around the failed replica automatically.
• Confidence that the remaining replicas can actually absorb the load of the failed one without themselves becoming overloaded and cascading the failure further.

That last point is often missed: if 3 servers are each running at 90% capacity and one dies, the other 2 now need to absorb that load — if they can't, you haven't tolerated the fault, you've just delayed and possibly widened it into a bigger outage.

Building in fault tolerance: pros and cons

Pros

• ✓A single component failing stays invisible to the user
• ✓No single failure can cascade into a full outage
• ✓Higher real-world reliability than "we just have a backup"

Cons

• ✕Extra idle/redundant capacity costs money even when nothing is failing
• ✕More operational complexity — isolation, timeouts, circuit breakers to maintain
• ✕Only proven to work if you actually test failure paths (chaos engineering)

Further Reading

• Wikipedia — Fault tolerance — a broad overview spanning software, hardware, and systems engineering definitions.
• Netflix Tech Blog — The Netflix Simian Army — the original description of Chaos Monkey and related tools built to continuously verify fault tolerance in production rather than assume it.