Skip to main content

Design a Distributed Message Queue (Kafka-like)

Every problem so far that needed asynchronous communication has treated Message Queues as a given piece of infrastructure to build on top of. This problem is the reverse: build the queue itself. It's a genuine step up in difficulty because a real message queue has to combine three things this course has so far treated mostly separately — ordered, durable storage; horizontal partitioning; and replication for fault tolerance — all in one system. Following the framework from How to Answer a System Design Interview Question.

1. Requirements​

Functional:

  • Producers append messages to a named topic; consumers read messages from a topic, each at their own pace, tracked by position.
  • Messages within a given ordering key must be delivered in the order they were produced.

Non-functional:

  • Must survive a broker (server) failure without losing acknowledged messages.
  • Must scale horizontally — a single topic's throughput can exceed what one machine can accept or store.
  • High sustained write throughput matters more than any single write's latency.

Scale estimate: assume 2 million messages/sec sustained across all topics, each message averaging 1KB (~2GB/sec of write throughput) — a volume that immediately rules out "one machine holds the whole topic" and makes partitioning a first-order design requirement, not an optimization.

2. API Design​

produce(topic, key, value) -> { partition, offset }
consume(topic, partition, from_offset) -> [{ offset, key, value }]
commit_offset(consumer_group, topic, partition, offset)

offset — a message's position within its partition — is the core primitive consumers track to know what they've already read, deliberately not a queue that deletes a message once delivered.

3. Data Model​

The core structure is an append-only log, not a table:

topic "orders" — split into partitions
  partition 0: [msg@0, msg@1, msg@2, ...]   -- an ordered, immutable sequence
  partition 1: [msg@0, msg@1, msg@2, ...]
  partition 2: [msg@0, msg@1, msg@2, ...]

Each partition is physically an append-only file (or set of files): writes are always appended at the end, never modified in place, which is what makes writes extremely fast (sequential disk I/O, no seeking) and lets many consumers read the same log independently without interfering with each other or with new writes.

4. High-Level Design​

System Design Lab

A consumer group is what turns this into the "many jobs, a pool of workers" pattern Message Queues describes generally: each partition is consumed by exactly one member of a given group at a time, so the group as a whole processes every partition, load-balanced across its members, while a different consumer group can independently read the entire topic again from the start — the same message log serving both "distribute this work" and "notify every independent subscriber" simultaneously, depending on how consumers are grouped.

5. Deep Dive: partitioning and why ordering is per-partition, not per-topic​

Splitting a topic into partitions is a direct application of Database Sharding: a producer's message key is hashed (the same Consistent Hashing-style idea) to deterministically pick which partition it lands in, so every message with the same key (e.g., all events for one order_id) always lands on the same partition, in the order they were produced.

This is precisely why ordering is guaranteed only within a partition, never across an entire topic: two messages with different keys can land on different partitions and be consumed in either order relative to each other, with no way to fix that without giving up partitioning's whole throughput benefit. This is a detail worth stating explicitly and precisely in an interview — a design that assumes topic-wide ordering when the system only actually guarantees per-partition ordering is a subtle, real correctness bug.

6. Deep Dive: replication and staying durable through a broker failure​

Each partition's log is replicated to multiple brokers — one leader accepting all writes for that partition, and follower brokers continuously replicating from it, exactly the Data Replication pattern applied per-partition instead of per-database. A message is only acknowledged to the producer once it's been written to a defined in-sync replica set (not just the leader alone), so a leader dying immediately after acknowledging a write doesn't lose that write — a surviving in-sync follower already has it.

System Design Lab

When a leader does fail, the remaining in-sync followers need to agree on exactly one replacement — a consensus problem, solved the same way Database Architectures describes for database leader failover: a majority-based election, not each follower independently deciding to promote itself.

7. Tradeoffs​

More partitions means more parallelism — more producers and consumers can work simultaneously without contending on the same log — but each additional partition is also more replicated state to manage and more overhead during a rebalance (when consumer group membership changes and partitions need reassigning among the remaining members). A topic over-partitioned far beyond what its actual throughput needs mostly just adds coordination overhead for no real benefit.

The acknowledgment rule from the replication deep dive above — wait for the in-sync replica set, not just the leader — is itself a tunable tradeoff, not a fixed choice. Requiring every in-sync replica to confirm a write before acknowledging the producer gives the strongest durability guarantee, at the cost of write latency bounded by the slowest in-sync replica, and a brief inability to accept writes at all if too many replicas fall out of sync simultaneously. Acknowledging as soon as the leader alone has written the message is faster and stays available through more failure scenarios, but reintroduces exactly the risk this design set out to close: a leader that dies immediately after acknowledging, before any follower replicated the write, loses that message for good.

More partitions vs. fewer partitions per topic: pros and cons​

More partitions

  • Higher achievable parallelism across producers and consumers
  • Finer-grained load distribution across brokers
  • More consumer group members can be usefully added later without a redesign

Fewer partitions

  • Caps how many consumers in a group can do useful parallel work at once
  • Less headroom to add consumers later without a repartitioning effort
  • Can under-utilize available broker capacity for a genuinely high-throughput topic

Further Reading​

Share this lesson

Saved locally in your browser — visible in the sidebar as you go.