[KIP-848] The Next Generation of the Consumer Rebalance Protocol
Here’s a rundown of KIP-848: The Next Generation of the Consumer Rebalance Protocol, which reached GA in Kafka 4.0. Its two central goals: cut rebalance downtime to nearly zero when consumer group membership changes, and move most of the responsibility for rebalancing from the client to the broker. Background The consumer group rebalancing protocol had been around for eight years, and it was running into structural limits. The biggest one was a thick-client design that put too much responsibility on the client. When a bug showed up in consumer group rebalancing, fixing it meant fixing the client, and if you’re running a cloud service, you can’t force your users to patch their own clients. Since most of the logic ran on the client side, diagnosing problems from server-side logs alone was often impossible. ...