Heartbeats
A heartbeat is a small message a node sends periodically to say, in effect, "I'm still alive." It's the most basic building block of failure detection in a distributed system: everything from a database promoting a new leader to a load balancer pulling a dead server out of rotation ultimately depends on something noticing a node has stopped responding — and a heartbeat is the simplest possible way to notice.
Service Discovery
In a system with a handful of services and a static IP address for each, one service can just hardcode where another one lives. That stops working the moment instances scale up and down, get rescheduled to different machines, or fail and get replaced — the exact address of "the orders service" is now constantly changing, and every caller needs a way to find the current, correct set of addresses without being redeployed every time it shifts. Service discovery is that mechanism: a way for services to find each other's current network location dynamically instead of by hardcoded configuration.
Consensus Algorithms
Consensus is the problem of getting multiple nodes to agree on a single value — even when some nodes might be slow, crash, or become unreachable — and having every node that does agree end up with the same answer. It sounds abstract, but it's the problem underneath several very concrete things covered elsewhere in this course: which replica becomes the new database leader, which node holds a distributed lock, what the current, authoritative state of a cluster's membership is.
Distributed Locking
A lock in a single process is easy: one piece of memory, one owner at a time, enforced by the operating system. A distributed lock solves the same problem — making sure only one actor can hold a resource at a time — across multiple independent machines that can't share memory and can't fully trust the network between them. It's the mechanism behind questions like "how do we make sure only one instance of this cron job actually runs" or "how do we stop two servers from both thinking they own this task."
Gossip Protocol
A gossip protocol spreads information across a cluster the same way a rumor spreads through a group of people: each node periodically picks a few random peers and shares what it currently knows, those peers do the same with their own random peers, and within a small number of rounds, information that started at one node has propagated to the entire cluster — with no central coordinator, and no node needing to know about every other node directly.
Circuit Breaker
When a downstream service starts failing, the naive behavior — every caller keeps retrying every request against it — makes things worse, not better it wraps calls to a dependency, watches for failures, and once failures cross a threshold, it stops sending traffic to that dependency entirely for a while — failing fast instead of piling on.
Disaster Recovery
Every failure mode covered so far in this module — a dead node, a failing dependency, a partitioned cluster — assumes the rest of the system is still there to react. Disaster recovery is what happens when the failure is bigger than that: an entire datacenter loses power, an entire cloud region goes offline, or data is corrupted or deleted at a scale no single-node failover can fix. It's the plan for "what if the thing we usually failover to is also gone."
Distributed Tracing
A single request in a microservices system rarely stays within one service — it might hit an API gateway, call three backend services, each of which queries its own database or calls another service in turn. When that request is slow, or fails, "which of these dozen hops was the problem?" is not a question logs on any one service can answer alone, because no single service's logs show the whole journey. Distributed tracing solves exactly this: it follows one request across every service it touches and reconstructs the full picture.