Load Balancing

A load balancer distributes incoming requests across multiple servers so that no single one is overwhelmed. It's one of the most consistently reused components across system design interviews, because it's the concrete mechanism behind several ideas already introduced in this course: it's how horizontal scaling actually spreads load in practice, it's usually where failover is implemented (health checks removing a dead server from rotation), and putting one in front of a single server is the standard first fix for the most common single point of failure in a naive design.

Layer 4 vs. Layer 7 load balancing

Load balancers operate at one of two levels of the OSI Model, and the choice has real consequences:

• Layer 4 (transport) load balancers route based on IP address and port alone, without looking at the request's actual content. They're fast and protocol-agnostic — they don't need to understand HTTP, or even that the traffic is HTTP at all — because they're just forwarding packets based on connection info.
• Layer 7 (application) load balancers terminate the connection and inspect the actual request — URL path, headers, cookies — before deciding where to route it. This unlocks content-aware routing (/api/* to one pool, /static/* to another) and features like sticky sessions via cookies, at the cost of more CPU work per request and slightly higher latency, since the proxy has to actually parse the application-layer protocol.

This is the exact same tradeoff that shows up when comparing a plain reverse proxy to an API gateway — more intelligence at the routing layer costs more per-request work.

Routing algorithms

Once a load balancer decides that it's routing to a pool of servers, it still needs a policy for which one:

Algorithm	How it decides	Good for
Round robin	Cycles through servers in order	Simple, roughly uniform request cost
Weighted round robin	Round robin, but bigger servers get proportionally more	Heterogeneous server capacity
Least connections	Sends to whichever server currently has the fewest active connections	Requests with widely varying duration
IP hash / consistent hashing	Same client (by IP or key) always maps to the same server	Session affinity, cache locality — see Consistent Hashing
Random	Picks a server at random	Cheap to compute, works surprisingly well at scale

Consistent hashing deserves a callout here specifically because load balancing is its most common real-world application: routing the same client (or the same cache key) to the same backend every time means that server's local cache stays warm for that client, instead of every request hitting a cold cache on a randomly chosen node.

Health checks tie it to failover

A load balancer is only as good as its ability to notice a server has died. It periodically pings each backend (or checks a dedicated health endpoint), and stops routing to any server that fails enough consecutive checks — this is the Failover mechanism from a client's point of view, just implemented inside the load balancer instead of via DNS or a promoted standby.

System Design Lab

Beyond a single load balancer: global load balancing

A single load balancer instance is itself a single point of failure, so real deployments run at least two behind a floating/virtual IP, or rely on DNS-based load balancing (round-robin or geo-DNS) one level up to spread traffic across multiple regional load balancers in the first place. This layered approach — DNS picks a region, a regional load balancer picks a server — is also the standard shape of a Disaster Recovery setup, where an entire region's load balancer can be removed from DNS rotation if the region goes down.

Why this matters in an interview

Naming the layer (L4 vs. L7) and the algorithm, not just "we'll add a load balancer," turns a generic answer into a specific engineering decision. For example: "we'll use an L7 load balancer with least-connections, since request durations vary a lot in this workload" says something a plain box-and-arrow diagram doesn't — and it's exactly the kind of specificity covered more generally in Vertical vs. Horizontal Scaling, where a load balancer is the piece of infrastructure that makes horizontal scaling actually usable.

Choosing Layer 7 over Layer 4 load balancing: pros and cons

Pros

• ✓Can route on URL path, headers, or cookies — enabling content-aware routing and sticky sessions
• ✓Can inspect and reject malformed or malicious requests before they reach a backend
• ✓Enables TLS termination and centralized logging at the proxy layer

Cons

• ✕Higher latency and CPU cost per request — the proxy must parse the application protocol
• ✕Protocol-specific — an L7 load balancer built for HTTP can't transparently balance arbitrary TCP traffic
• ✕More configuration surface area (routing rules, header inspection) than a simple L4 forwarder

Further Reading

• AWS — What is Load Balancing? — a practical overview of load balancer types and algorithms as implemented in cloud infrastructure.
• NGINX Docs — HTTP Load Balancing — a hands-on guide to configuring routing algorithms and health checks in a real load balancer.