Distributed Systems

A Kafka partition has always been limited to one consumer at a time. That coupled partition count and consumer count tightly together, and scaling throughput often meant splitting a topic into far more partitions than the data itself justified. Plenty of workloads genuinely fit a queue-style model better, where several consumers split the work on one partition, but the old structure had no way to express that. KIP-932: Queues for Kafka removes that constraint with a new group type: the Share Group. ...

Here’s a rundown of KIP-848: The Next Generation of the Consumer Rebalance Protocol, which reached GA in Kafka 4.0. Its two central goals: cut rebalance downtime to nearly zero when consumer group membership changes, and move most of the responsibility for rebalancing from the client to the broker. Background The consumer group rebalancing protocol had been around for eight years, and it was running into structural limits. The biggest one was a thick-client design that put too much responsibility on the client. When a bug showed up in consumer group rebalancing, fixing it meant fixing the client, and if you’re running a cloud service, you can’t force your users to patch their own clients. Since most of the logic ran on the client side, diagnosing problems from server-side logs alone was often impossible. ...

This post walks through KIP-101 — Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation, the proposal that introduced the leader epoch to Kafka, along with some notes of my own. Log Replication Protocol Before we get to the leader epoch, let’s look at how a partition log replicates from the leader broker to its followers. The exact flow shifts a bit depending on the producer’s ack setting and min.isr, but I want to focus on the core mechanics here. ...