Skip to main content

Distributed Caching

A single cache server has the same ceiling any single machine does: finite memory, finite throughput, and it's a Single Point of Failure — if it goes down, every request behind it falls straight through to the database at once. Distributed caching spreads the cache itself across multiple machines, applying the same horizontal scaling idea already covered for application servers and databases to the caching layer.

Local cache vs. distributed cache​

It's worth being precise about a distinction that's easy to blur: a local (in-process) cache lives in one application server's own memory — fastest possible access, but every server has its own separate copy, so a value cached on server A is invisible to server B, and N servers each independently caching the same data multiply memory usage by N. A distributed cache is a separate, shared tier (e.g. a Redis or Memcached cluster) that every application server talks to over the network — one shared copy, consistent across all callers, at the cost of a network hop that a local cache doesn't pay.

Many real systems layer both: a small, very fast local cache for extremely hot data, backed by a distributed cache for everything else, which is itself backed by the database — a small tower of increasingly large, increasingly slower tiers.

Splitting the cache itself: back to consistent hashing​

Once the cache is bigger than one machine can hold, keys need to be assigned to specific cache nodes — the exact same problem Database Sharding solves for databases, and it's solved the same way: Consistent Hashing hashes both keys and nodes onto a shared ring, so adding or removing a cache node only remaps a small fraction of keys instead of invalidating the entire cache at once. This is in fact the scenario Consistent Hashing's own lesson names as one of its primary real-world motivations — distributed cache client libraries were among the first popularizers of the technique, specifically to avoid a "cache miss storm" every time the cluster resized.

System Design Lab

Keeping a distributed cache correct: invalidation across nodes​

A local cache only ever needs to worry about staleness relative to the database. A distributed cache adds a second dimension: when data changes, every node (or every application server's local cache) that might be holding a stale copy needs to find out — a write on one server can't just clear its own view and call it done, because other servers may still be serving the old value. Common approaches include a short TTL as a simple upper bound on staleness (accept some inconsistency, bound how long it can last), or an explicit invalidation message broadcast to all cache nodes when a write happens (more immediately correct, more moving parts). This is a direct instance of the same problem Data Replication solves for databases — keeping multiple copies of the same data acceptably in sync — just applied to a cache instead of a primary data store.

Two failure modes distributed caches specifically add​

  • A node failure loses only its slice, not everything. This is the direct payoff of sharding the cache: unlike a single cache instance going down (which sends 100% of traffic straight to the database), losing one node in a distributed cache only turns that node's keys into misses, while the rest of the cache keeps absorbing load — a much softer failure.
  • Hot keys can still overwhelm a single node. Consistent hashing spreads keys evenly on average, but one extremely popular key (a viral post, a celebrity's profile) still lives on exactly one node — sharding the keyspace doesn't help when the traffic is this lopsided. This is usually addressed by additionally replicating especially hot keys across multiple nodes, or falling back to a local cache specifically for the small set of keys hot enough to need it.

Why this matters in an interview​

Naming consistent hashing as the mechanism for distributing cache keys — rather than a vague "we'll cluster the cache" — connects this lesson directly back to a technique already covered earlier in the course, and is exactly the kind of specific, checkable detail that reads as fluency. It's also worth proactively naming how invalidation propagates across nodes, since "the cache is distributed" raises the staleness question again in a slightly harder form than a single cache does.

Distributed cache vs. local (in-process) cache: pros and cons​

Distributed cache

  • One shared, consistent copy across every application server
  • Total cache capacity scales with the cluster, not one machine's memory
  • Losing one node only affects its slice of keys, not the whole cache

Local cache

  • Each server duplicates cached data, multiplying total memory usage by server count
  • One value cached on server A is invisible to server B — no shared view
  • Not horizontally scalable — capacity is capped by a single machine's memory

Further Reading​

Saved locally in your browser — visible in the sidebar as you go.