Latency vs Throughput vs Bandwidth

These three words get used loosely in everyday conversation, but in a system design interview they mean three distinct, measurable things — and conflating them is one of the fastest ways to sound imprecise.

• Latency is how long a single operation takes, from request to response. Measured in time (milliseconds, seconds).
• Throughput is how many operations a system completes per unit of time. Measured in operations per second (requests/sec, transactions/sec).
• Bandwidth is the maximum rate at which data can be transferred across a link. Measured in bits or bytes per second (Mbps, Gbps).

A simple mental model: imagine a highway. Latency is how long it takes one car to drive from one end to the other. Throughput is how many cars per minute arrive at the far end. Bandwidth is how many lanes the highway has — it's the ceiling on throughput, not a measurement of any single car's trip.

Why throughput isn't just "1 / latency"

If every request were handled strictly one at a time, throughput would indeed just be the inverse of latency. But real systems handle requests concurrently — while one request is waiting on a disk read or a network call, the CPU can work on another. That's why a system can have both individually-slow requests and high throughput, as long as enough of them can be in flight at once (this is the whole idea behind async I/O and connection pooling).

The complication is that concurrency has a limit. Every system has some finite capacity — CPU cores, database connections, thread pool size — and once you push more concurrent work at it than that capacity allows, requests start queueing. Queued requests wait before they even start being processed, which shows up as rising latency, while throughput stops increasing (or even drops slightly, since the system now spends resources on context-switching and managing the queue instead of doing useful work). This saturation pattern shows up so often it's worth seeing plotted:

Throughput vs. load

Latency vs. load

View underlying data

Concurrent requests	Throughput (req/s)	Latency (ms)
10	40	6
20	80	8
30	120	13
40	160	25
50	196	60
60	196	130
70	194	210
80	190	300
90	185	420
100	178	560

Illustrative data for a hypothetical fixed-capacity server, not a measurement. Notice throughput levels off (and slightly degrades) once the system saturates around 50 concurrent requests, while latency keeps climbing — this is why "requests per second" alone never tells the whole story.

This is why "requests per second" alone is an incomplete way to describe capacity — a system can technically process 200 req/s, but if it only does that at a latency of half a second per request, most user-facing products would already consider it degraded well before that theoretical ceiling.

Why this distinction matters in an interview

When you're asked to estimate capacity for a system, you're implicitly being asked about throughput ("how many requests per second do we need to support?"). When you're asked whether users will find the system usable, you're being asked about latency ("how long does a user wait for their feed to load?"). And when you're moving large payloads — video, backups, bulk exports — bandwidth becomes the binding constraint, independent of how fast any single server can compute a response.

A concrete example of why all three matter together: a video streaming service could have very low latency to start a request and enormous throughput in requests handled per second, but if the network link to a user's device doesn't have the bandwidth to sustain the video's bitrate, playback still stutters. No amount of server-side latency or throughput optimization fixes a bandwidth-constrained link.

A rule of thumb for interviews

When you propose scaling a component, say explicitly which of the three you're optimizing for scaling for. "Adding more application servers behind a load balancer increases throughput" is a much stronger sentence than "adding more servers makes it faster" — the second one is ambiguous about whether you mean individual request latency (usually unaffected, or even slightly worse due to added network hops) or aggregate throughput (usually what actually improves).

Optimizing for throughput over latency: pros and cons

Pros

• ✓Higher overall system capacity for the same hardware
• ✓Better utilization — fewer idle CPU cycles between requests
• ✓Lower infrastructure cost per request at scale

Cons

• ✕Individual requests can sit queued longer under load
• ✕Tail latency (p99) suffers even while average throughput looks great
• ✕Users perceive the "slow" queued requests, not the healthy aggregate number

Further Reading

• Latency Numbers Every Programmer Should Know — an interactive, frequently cited visualization comparing the latency of memory, disk, network, and cross-region operations.
• Cloudflare Learning Center — Latency vs Throughput — a clear, practitioner-oriented explanation of the two terms and how bandwidth relates to both.