Rate Limiting Explained: Token Bucket and Beyond
Rate limiting is a deliberately blunt tool: instead of trying to make every request cheap, it accepts that some requests get turned away, so the system stays up for everyone else.
Not the same job as load balancing
It's worth being precise about the difference: load balancing assumes the incoming traffic is legitimate and spreads it out efficiently. Rate limiting assumes some incoming traffic might be excessive or abusive, and its job is to say no to some of it — typically with an HTTP 429 Too Many Requests — before it ever reaches real business logic.
Why a service needs this at all
Without a limit, one misbehaving client — a buggy retry loop, a scraper, a deliberate abuser — can consume a disproportionate share of a service's capacity, degrading it for everyone else. It's the standard first line of defense against basic denial-of-service traffic, and a practical necessity for metering usage on any API with paid tiers. Rate limiting is typically enforced once, centrally, at an API gateway — so individual backend services don't each have to reimplement the same logic, and a request that's going to be rejected anyway never costs the backend any real work.
The algorithms
| Algorithm | How it works | Tradeoff |
|---|---|---|
| Fixed window | Count requests in a fixed clock interval, reset each interval | Simple, but allows a burst of 2x the limit right at a window boundary |
| Sliding window | Count requests in a rolling window ending "now" | Smooths the boundary-burst problem, costs more to compute |
| Token bucket | A bucket refills at a steady rate; each request spends one token | Allows controlled bursts up to bucket size, enforces a true average rate |
| Leaky bucket | Requests queue and drain at a fixed steady rate | Smooths bursts into steady output, adds queueing latency |
System Design LabWhy token bucket is the go-to answer
Token bucket is the most commonly cited algorithm in interviews because it fits how real traffic actually behaves: legitimate clients are often bursty — a page load firing off several requests at once — and token bucket tolerates that burst as long as the average rate stays within budget. Fixed window, by contrast, either lets through a burst it shouldn't (right at a window boundary) or blocks a burst that was actually fine.
What key you limit on matters as much as the algorithm
The algorithm decides how to count; the key decides who's being counted. Per-IP limiting can unfairly throttle many legitimate users sitting behind the same corporate NAT or proxy, while per-user or per-API-key limiting is more precise but only works once a client is authenticated — which is why public, unauthenticated endpoints often fall back to IP-based limiting as a coarser first line of defense.
Tell the client where it stands
A well-designed rate-limited API doesn't just reject silently at the threshold — it signals via response headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, so a well-behaved client can back off proactively instead of hammering the API until it gets a 429. It's a small detail, but naming it signals you're thinking about the client's experience of the limit, not just the server's enforcement of it.
Why this matters in an interview
"We'll add rate limiting" is weak on its own. Naming the algorithm (token bucket, for bursty-but-bounded traffic), where it's enforced (the gateway, so rejected requests cost the backend nothing), and what key it's keyed on turns it into a specific, defensible decision. It's also the natural complement to a circuit breaker — rate limiting protects a service from too much legitimate demand coming in, while a circuit breaker protects callers from a dependency that's already failing going out.
Go deeper
The full lesson expands on each algorithm's boundary behavior and where rate limiting fits inside an API gateway:
👉 Read the full Rate Limiting lesson — part of the free System Design Lab course.