Heartbeats

A heartbeat is a small message a node sends periodically to say, in effect, "I'm still alive." It's the most basic building block of failure detection in a distributed system: everything from a database promoting a new leader to a load balancer pulling a dead server out of rotation ultimately depends on something noticing a node has stopped responding — and a heartbeat is the simplest possible way to notice.

The mechanism is almost embarrassingly simple

A node (or a dedicated monitor) sends a tiny "I'm alive" signal to its peers on a fixed interval. Peers track the last time they heard from each node, and if that gap exceeds some timeout, they consider the node dead and act accordingly — this is exactly the health-check mechanism briefly introduced in Failover and Load Balancing, spelled out as its own primitive.

System Design Lab

The tradeoff hiding in one number: the timeout

Every heartbeat-based system has exactly one knob that matters most, and it's a direct latency-vs-correctness tradeoff:

Too short a timeout — a node that's merely slow (a garbage-collection pause, a brief network hiccup, a CPU spike) gets falsely declared dead, triggering an unnecessary and potentially disruptive failover for a node that was actually fine.
Too long a timeout — a genuinely dead node keeps being treated as alive for longer, meaning requests keep getting routed to it (or a leader election keeps being delayed) well after it actually failed.

There's no universally correct value — it's a direct trade of detection speed against false-positive risk, tuned to what a specific system can tolerate. This is the same shape of tradeoff CAP Theorem forces at a higher level: deciding how long to wait for confirmation before acting is fundamentally a bet about what's actually happening on an unreliable network you can't directly observe.

Heartbeats don't scale to "everyone pings everyone"

A naive design — every node sending a heartbeat directly to every other node — costs O(n²) messages as the cluster grows, which becomes impractical well before a cluster gets very large. Real systems avoid this in one of two ways: a centralized monitor that every node reports to (simple, but reintroduces a Single Point of Failure unless the monitor itself is made redundant), or a decentralized, epidemic-style approach where liveness information spreads peer-to-peer instead of through direct all-to-all pings — the approach covered in full in Gossip Protocol, which uses heartbeat information as exactly the kind of state it disseminates.

What a heartbeat's "dead" verdict actually triggers

A heartbeat timeout by itself doesn't do anything — it's the input to some other mechanism that reacts to it, and naming that mechanism is what turns "the system detects failures" into a real design:

In a leader-based database, a missed heartbeat from the leader triggers a new leader election, which is a consensus problem in its own right — the replicas have to agree on exactly one replacement, not each independently promote themselves.
In a load balancer, a missed heartbeat from a backend removes it from the routing pool, the same health-check behavior already covered in Load Balancing.
In a service registry, a missed heartbeat deregisters an instance so it stops being handed out to callers — the mechanism Service Discovery relies on to keep its registry accurate.

Why this matters in an interview

"The system detects failed nodes via heartbeats" is a good start, but naming the timeout tradeoff explicitly — and what happens after a heartbeat is missed — is what separates a real answer from a buzzword. A strong answer also proactively distinguishes "the node is dead" from "the node is unreachable from here but alive elsewhere," a distinction with no clean resolution on an asynchronous network, which is exactly the kind of nuance worth surfacing rather than glossing over.

Short vs. long heartbeat timeouts: pros and cons

Short timeout

Detects a genuine failure quickly, minimizing time spent routing to a dead node
Triggers failover sooner, reducing the window of degraded availability
Better fit for latency-sensitive systems where staleness is costly

Long timeout

Tolerates a truly dead node for longer before anything reacts
Delays failover, extending the window where requests may fail or stall
Only justified when false-positive failovers are more costly than slower detection

The mechanism is almost embarrassingly simple​

The tradeoff hiding in one number: the timeout​

Heartbeats don't scale to "everyone pings everyone"​

What a heartbeat's "dead" verdict actually triggers​

Why this matters in an interview​

Short vs. long heartbeat timeouts: pros and cons​

Further Reading​