Skip to main content

Reliability

Reliability is the probability that a system performs its intended function correctly over a given period of time, without failing. Where availability asks "is it up right now?", reliability asks a stricter question: "does it keep doing the right thing, consistently, for as long as it's expected to?"

This distinction shows up constantly in real systems. A payment service that's up 99.99% of the time but occasionally double-charges a customer during a retry is highly available and unreliable. An overnight batch job that runs correctly every single time it executes, but only runs once a day (so a bug takes 24 hours to notice and there's no redundancy if the one run fails), can be considered reliable in a narrow sense but would never be described as highly available.

What makes a system unreliable​

Failures tend to come from a fairly small set of recurring causes, and naming them specifically is more convincing in an interview than saying "things could break":

  • Hardware failure — disks die, memory corrupts, network links flap. At large enough scale, hardware failure isn't an edge case, it's a constant background rate you have to design around.
  • Software bugs — a null pointer, an unhandled edge case, a race condition that only shows up under specific timing.
  • Human error — a bad deploy, a misconfigured load balancer, an accidentally deleted table. In mature systems, human error is often the single largest cause of outages, which is why safeguards like staged rollouts, code review, and the ability to quickly roll back matter as much as any purely technical design choice.
  • Cascading failure — one component's failure increases load or latency on its neighbors, which then fail too, spreading a small problem into an outage. This is why patterns like Circuit Breaker and sensible timeouts exist: to stop failures from propagating.
System Design Lab

Designing for reliability​

You can't eliminate failure, so reliable systems are built around the assumption that failure will happen, and designed to survive it gracefully rather than avoid it entirely. Some of the recurring techniques:

  • Redundancy — no component that matters should exist as a single copy. This is the same idea that underlies Fault Tolerance and directly attacks Single Points of Failure.
  • Replication — keep multiple up-to-date copies of data so losing one copy (or one machine) doesn't lose the data. See Data Replication.
  • Health checks and monitoring — you can't respond to a failure you don't know is happening. Reliable systems detect problems (often before users notice) through heartbeats, error-rate alerts, and synthetic checks.
  • Idempotency and retries — if a client can safely retry an operation without causing duplicate effects, transient failures (a dropped connection, a momentarily overloaded server) become invisible to the user instead of becoming errors. See Idempotency.
  • Testing failure on purpose — practices like chaos engineering (deliberately killing servers or injecting latency in production, pioneered publicly by Netflix's Chaos Monkey) exist because the only reliable way to know a system tolerates a failure mode is to actually trigger it in a controlled way, rather than assuming the design handles it.

Measuring reliability​

Two metrics come up repeatedly:

  • MTBF (Mean Time Between Failures) — the average time a system runs before it fails. Higher is better.
  • MTTR (Mean Time To Recovery/Repair) — the average time it takes to restore service after a failure. Lower is better.

Interestingly, a lot of modern reliability engineering focuses more on driving MTTR down than MTBF up — because at large scale, some failures are effectively unavoidable, so a system that fails occasionally but recovers in seconds (via automated failover) can deliver better real-world reliability than one that fails rarely but takes hours to fix by hand when it does.

Investing in reliability engineering: pros and cons​

Pros

  • Fewer production incidents and less time spent firefighting
  • Predictable, trustworthy behavior under real-world failure conditions
  • Faster recovery (lower MTTR) when something does go wrong

Cons

  • Slower feature velocity — time spent on resilience isn't spent on features
  • Higher upfront engineering cost (retries, health checks, chaos testing)
  • Requires organizational buy-in; easy to under-invest until an outage forces the issue

Further Reading​

Saved locally in your browser — visible in the sidebar as you go.