Skip to main content

Caching 101

A cache is a copy of data kept somewhere faster or cheaper to access than the place that data actually lives, so that a repeated request can be answered from the copy instead of redoing the original expensive work. It's arguably the single highest-leverage tool in system design, precisely because it doesn't require changing the underlying system at all — a cache sits in front of something slow and simply intercepts requests before they get there.

Hits, misses, and why the gap matters​

Every cache lookup is either a hit (the data was found in the cache, and the expensive path was skipped) or a miss (it wasn't, so the request falls through to the real source — a database query, a computed result, a call to another service — and the result is typically written into the cache on the way back, so the next request for it is a hit).

System Design Lab

The entire value of a cache is captured by its hit rate — the fraction of requests answered from the cache — multiplied by how much cheaper a hit is than a miss. A cache in front of a database query that takes 200ms, hit 95% of the time, turns the average request cost into roughly 10ms of cache lookup plus a small tail of full-cost misses — the same order-of-magnitude win that an index gives a single query, but applied in front of the database rather than inside it. The two aren't competing: an index makes the miss path itself faster, while a cache reduces how often the miss path runs at all.

Where a cache can live​

Caching isn't one specific technology — it's a pattern that shows up at nearly every layer a request passes through:

  • Client-side — a browser caching a static asset, or a mobile app caching the last-fetched screen of data, so the network is skipped entirely on a hit.
  • CDN / edge — caching content geographically close to the user, cutting the latency of the trip itself; this gets a full lesson in CDN.
  • Application layer — an in-memory cache (or a dedicated cache server like Redis or Memcached) sitting between application code and the database.
  • Database layer — many databases keep their own internal buffer cache of recently-accessed pages in memory, transparently to the application.

The further "up" (closer to the client) a cache lives, the more it saves — a client-side hit skips the network entirely, while a database-layer hit still pays for a request all the way there. But caches closer to the client are also harder to keep correct, which is exactly the tension the rest of this module works through.

The one problem every cache eventually has: staleness​

A cache is, by definition, a copy — and a copy can go out of date the moment the original changes. This is the central, unavoidable tradeoff of caching, memorably summarized by the industry saying "there are only two hard things in computer science: cache invalidation and naming things." A cache makes reads faster at the cost of introducing a window where a client can see data that's no longer accurate, and every caching design has to make a deliberate choice about how long that window is allowed to be, and how the cache finds out something changed. Caching Strategies covers the specific patterns (cache-aside, write-through, TTLs) for managing exactly this.

When caching isn't the answer​

Caching helps most when the same data is read far more often than it changes — a product page, a user's profile, a computed leaderboard. It helps far less for data that's read once and rarely repeated, or that changes on every write (caching it would mean invalidating it almost immediately, for little benefit). Recognizing which reads in a design are actually repeated enough to be worth caching is as important as knowing how caching works at all.

Why this matters in an interview​

"We'll add a cache" is a fine instinct, but a strong answer says what specifically gets cached, why that data is read far more than it's written, and what happens when the underlying data changes — that last question is the one most candidates skip, and it's exactly the one this whole module is built to answer.

Adding a cache in front of a slow data source: pros and cons​

Pros

  • Can turn a request that took hundreds of milliseconds into a single-digit-millisecond lookup
  • Reduces load on the underlying database or service, not just latency for the requester
  • Doesn't require changing the underlying system — it sits in front of it

Cons

  • Introduces a window where cached data can be stale relative to the source of truth
  • Adds a new component (and a new failure mode) to operate and monitor
  • Only helps for data that's actually read repeatedly — no benefit for one-off reads

Further Reading​

Saved locally in your browser — visible in the sidebar as you go.