Rate Limiting

Rate limiting caps how many requests a client can make in a given window of time, and rejects the rest — typically with an HTTP 429 Too Many Requests — before they ever reach the actual business logic. It's a deliberately blunt tool: rather than trying to make every request cheap, it accepts that some requests will be turned away, in exchange for guaranteeing the system as a whole stays up for everyone else.

It's worth being precise about what problem this solves, because it's easy to conflate with load balancing: a load balancer assumes the incoming traffic is legitimate and spreads it out efficiently; a rate limiter assumes some incoming traffic might be excessive or abusive, and its job is to say no to some of it before it ever gets that far.

Why a service needs this at all

Without a limit, a single misbehaving client — a buggy retry loop, a scraper, or a deliberate abuse attempt — can consume a disproportionate share of a service's capacity, degrading it for every other client. This is directly a reliability and fault tolerance concern: rate limiting is what keeps one bad actor's traffic from becoming everyone else's outage. It's also the standard first line of defense against basic denial-of-service traffic, and a practical necessity for metering usage on any API with paid tiers.

Where it's enforced

Rate limiting is one of the textbook responsibilities centralized in an API Gateway: enforcing quotas once at the edge, before a request reaches any backend service, means individual services don't each need to reimplement the same logic — and a request that's going to be rejected anyway never costs the backend any real work.

The algorithms

Different algorithms trade off burstiness, precision, and implementation cost differently — naming the specific one is what turns "we'll rate limit it" into an actual engineering decision:

Algorithm	How it works	Tradeoff
Fixed window	Count requests in a fixed clock interval (e.g. per minute); reset the count each interval	Simple, but allows a burst of 2x the limit right at a window boundary
Sliding window	Count requests in a rolling window ending "now," not a fixed clock boundary	Smooths out the boundary-burst problem, more accurate, costs more to compute
Token bucket	A bucket holds tokens, refilled at a steady rate; each request consumes one token, and requests are rejected when the bucket is empty	Allows controlled bursts (up to the bucket size) while enforcing a steady average rate
Leaky bucket	Requests queue up and are processed (or dropped) at a fixed steady rate, regardless of how bursty the input is	Smooths bursts into a steady output rate, at the cost of added queueing latency

Token bucket is the most commonly cited answer in interviews specifically because it handles the realistic case well: legitimate clients are often bursty (a page load firing off several requests at once), and token bucket accommodates that burst as long as the average rate stays within budget — unlike fixed window, which either allows a burst it shouldn't (near a window boundary) or blocks a burst that was actually fine.

System Design Lab

What key to limit on

The algorithm decides how to count; the key decides who's being counted. Common choices are per-API-key, per-user-ID, or per-IP-address, each with a different failure mode worth naming: IP-based limiting can unfairly throttle many legitimate users sitting behind the same corporate NAT or proxy, while a well-authenticated per-user or per-API-key limit is more precise but only works once a client is authenticated — which is why public, unauthenticated endpoints often fall back to IP-based limiting as a coarser first line of defense.

Signaling limits back to the client

A well-designed rate-limited API doesn't just reject silently at the threshold — it tells the client where it stands, typically via response headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, so a well-behaved client can back off proactively instead of hammering the API until it gets a 429. This is a small detail, but naming it signals you're thinking about the client's experience of the limit, not just the server's enforcement of it.

Why this matters in an interview

"We'll add rate limiting" is a weak answer on its own; naming the algorithm (token bucket, for bursty-but-bounded traffic), where it's enforced (the gateway, so rejected requests never cost the backend anything), and what key it's keyed on (per-user vs. per-IP, and why) turns it into a specific, defensible design decision — exactly the kind of specificity this module has been building toward throughout.

Choosing token bucket over fixed window: pros and cons

Pros

• ✓Tolerates legitimate bursts up to the bucket size without extra logic
• ✓Enforces a true steady-state average rate over time
• ✓No sharp discontinuity in allowed traffic at a clock boundary

Cons

• ✕More state to track per client than a simple interval counter
• ✕Choosing the right bucket size and refill rate takes real tuning
• ✕Slightly more expensive to compute per request than a fixed-window count

Further Reading

• Cloudflare Learning — What is rate limiting? — a practical overview of why and where rate limiting is applied in real infrastructure.
• Stripe Engineering — Scaling your API with rate limiters — a detailed walkthrough of several rate-limiting algorithms as implemented in a real production API.