Questions
4 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 8/10
Coordination and rebalancing
Cooperative rebalancing moves only the partitions that must change owners, over multiple rounds, so unaffected consumers keep processing instead of stopping the world
With the eager protocol, every rebalance begins with all members revoking all of their partitions, rejoining, and receiving a fresh assignment. Even when only one partition needs to move, the whole group stops consuming during the rebalance, and consumers with in-memory state or caches lose them. Cooperative rebalancing (incremental, KIP-429, introduced in 2.4 with CooperativeStickyAssignor) changes the contract: members keep the partitions they will retain, and only partitions that must move are revoked. It takes two rounds. In round one, the leader computes the target assignment and each member revokes only the partitions it is losing; the revoked partitions are temporarily unowned. In round two, those freed partitions are assigned to their new owners. Consumers whose partitions did not move never stop processing.
The benefit is largest when the group is big, when processing is stateful (Kafka Streams state stores, local caches), or when rebalances are frequent, such as during rolling deploys. The costs are real: the total time to reach a final assignment can be longer because of the extra round, listener callbacks change meaning (onPartitionsRevoked and onPartitionsAssigned receive only the delta, and onPartitionsLost handles involuntary loss), and the migration needs care because eager and cooperative assignors cannot be mixed in a live group. The standard upgrade is a rolling two-bounce: first deploy a configuration listing both CooperativeStickyAssignor and the current eager assignor, so the group negotiates the common eager protocol; then, once all members are upgraded, remove the eager assignor in a second rolling deploy to switch to cooperative.
javascript
- 1
Trade-off: less interruption per rebalance, but more rounds and a more complex callback model. For small, stateless groups with rare rebalances, the gain may not justify migration risk.
- 2
Common mistake: assuming the default already gives cooperative behavior. Since 3.0 the default list is RangeAssignor then CooperativeStickyAssignor, and the group uses the first protocol every member supports, so it stays eager until Range is removed. Check your client version and config.
- 3
Common mistake: writing listener code that assumes onPartitionsAssigned receives the full assignment. In cooperative mode it is incremental, so use c.assignment() when you need the whole set.
- 4
Common mistake: expecting cooperative rebalancing to fix a rebalance storm. It reduces the cost of each rebalance, not how often they happen. Fix the triggers (poll interval, restarts, timeouts) first.
- 5
Limit: if most partitions move anyway (a big scale-in or total restart), the benefit shrinks. Sticky assignment minimizes movement but cannot avoid it when membership changes drastically.
- 6
Alternative: static membership (group.instance.id) avoids rebalances on quick restarts, and the two are complementary. Kafka Streams has its own task assignment and has used cooperative rebalancing by default in modern versions, but verify for your release.
- 7
Version note: the KIP-848 protocol (GA in Kafka 4.0) moves reconciliation to the broker and is incremental by design, so partition.assignment.strategy and eager versus cooperative stop being the relevant knobs for groups using it. Confirm which protocol your group runs.
Scenario Questions
0-2 years experience
2-5 years experience
- 1A consumer keeps an in-memory cache per partition. Explain how eager and cooperative rebalancing treat that cache differently when a new member joins.
- 2You are told to switch a running production group to the cooperative assignor. Describe the safe rollout steps and what goes wrong if you flip the config in a single deploy.
5-8 years experience
- 1After switching to cooperative rebalancing, a handler that flushed state in onPartitionsAssigned starts losing data. What changed in the callback contract and how do you fix it?
- 2Rebalances still occur every few minutes after adopting cooperative rebalancing. Explain why this is expected and what you would investigate instead.
8+ years experience
- 1Compare cooperative rebalancing, static membership and the KIP-848 protocol as ways to reduce rebalance disruption. For a 300-pod stateful consumer fleet doing daily rolling deploys, which combination would you choose and why?
- 2Quantify the trade-off: when could cooperative rebalancing make the total time-to-stable-assignment longer, and in which workloads would you therefore not adopt it?