Distributed Locking

A lock in a single process is easy: one piece of memory, one owner at a time, enforced by the operating system. A distributed lock solves the same problem — making sure only one actor can hold a resource at a time — across multiple independent machines that can't share memory and can't fully trust the network between them. It's the mechanism behind questions like "how do we make sure only one instance of this cron job actually runs" or "how do we stop two servers from both thinking they own this task."

What a distributed lock actually needs to guarantee

A distributed lock has to hold up under exactly the failure modes single-machine locks never face: a node can crash while holding the lock (who releases it?), a network partition can make a node look dead when it isn't (can it still believe it holds the lock?), and messages can be delayed arbitrarily (can a "stale" lock holder's write arrive after someone else has already taken over?). A correct implementation has to answer all three, not just the happy path of "one client asks, one client gets it."

System Design Lab

Built on top of consensus, not reinvented from scratch

Distributed locks are almost never implemented as a bespoke protocol — they're typically built on top of a system that already solves Consensus Algorithms, like ZooKeeper, etcd, or Redis with a properly implemented algorithm (Redlock). This isn't a coincidence: "who currently holds the lock" is exactly the kind of single, agreed-upon value a consensus system guarantees stays consistent even as nodes fail and network conditions change — reaching for a battle-tested coordination service is standard practice rather than building lock semantics directly on top of raw network calls.

The problem a lease/TTL solves — and the one it creates

If a client crashes while holding a lock and never releases it, a lock with no expiration is held forever, deadlocking every other client permanently. The standard fix is a lease: the lock is granted with a time-to-live, and it's automatically released if the holder doesn't renew it before the TTL expires. This directly parallels the timeout tradeoff from Heartbeats — too short a TTL risks releasing a lock out from under a client that's just running slowly (a garbage-collection pause, in the canonical example); too long a TTL means a genuinely crashed client's lock stays held, and everyone else stays blocked, for longer than necessary.

This creates a subtler correctness problem worth naming explicitly: a client can be paused (GC pause, scheduling delay) for longer than the TTL, have its lock silently expire and get reassigned to another client, and then resume and continue acting as if it still holds the lock — now two clients believe they own the same resource simultaneously, which is exactly the failure the lock was supposed to prevent. The standard mitigation is a fencing token: a monotonically increasing number issued with each lock grant, which the protected resource itself checks and rejects any write carrying an older token than one it's already seen — turning "the lock service says only one client should be acting" into a guarantee the resource itself enforces, rather than trusting clients to behave.

It's worth being precise about how this differs from Idempotency: idempotency makes it safe for an operation to run more than once by ensuring repeats have no additional effect; a distributed lock instead tries to prevent an operation from running more than once concurrently in the first place. They're complementary, not substitutes — a well-designed system facing "make sure this job only runs once" often uses a lock to avoid the common case of double-execution, and idempotency as the safety net for the rare case where locking still lets a duplicate slip through (a stale lock, a fencing gap).

Why this matters in an interview

Naming a distributed lock is the right move for "exactly one instance should do this" problems — but a strong answer goes further and names the crash-while-holding problem, proposes a TTL/lease as the fix, and then names the new problem that creates (a paused client acting past its lease) and fencing tokens as the standard resolution. That chain of reasoning is what distinguishes "we'll use a distributed lock" from a design that's actually been thought through under failure.