Questions
3 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 5/10
Coordination and rebalancing
A rebalance happens whenever group membership, the subscription, or the partition set changes, so partition ownership must be recomputed
A rebalance is the process of redistributing partitions among group members. Conceptually, the group's job is to keep the invariant that every subscribed partition has exactly one owner. Anything that breaks the current assignment forces recomputation. The triggers fall into three families: membership changes, subscription changes, and topic metadata changes.
- 1
Membership: a consumer joins (new pod, scale-out, restart), leaves cleanly (close() or shutdown), or is removed because the coordinator stopped hearing heartbeats within session.timeout.ms (crash, network partition, long GC or process freeze).
- 2
Poll liveness: a member that does not call poll() within max.poll.interval.ms proactively leaves the group, even though its heartbeat thread is healthy.
- 3
Subscription: a member changes its subscribed topics, or a regex subscription starts matching a newly created topic.
- 4
Partition metadata: partitions are added to a subscribed topic, or a subscribed topic is deleted or created.
- 5
Coordinator events: the group coordinator broker fails over, which can force members to rediscover it and rejoin.
In the classic protocol the sequence is: the coordinator marks the group as rebalancing, every member sends JoinGroup, the coordinator picks a group leader, the leader computes an assignment with the configured assignor, and SyncGroup distributes it. The group moves to a new generation, and commits from an old generation are rejected, which is how Kafka fences zombie consumers. With the default eager protocol all members revoke all partitions first, so processing stops group-wide during this window. That is why rebalances are expensive and why frequent ones are a symptom to investigate, not background noise.
javascript
- 1
Trade-off: a short session.timeout.ms detects crashes quickly but turns pauses and network blips into rebalances. A long one tolerates pauses but delays failover. Static membership (group.instance.id) lets a restarting instance keep its partitions without a rebalance, at the cost of slower removal of a truly dead member.
- 2
Common mistake: believing rebalances only happen when a consumer crashes. Scale-out, deploys, topic changes and slow poll loops all trigger them.
- 3
Common mistake: confusing session.timeout.ms (heartbeat thread liveness) with max.poll.interval.ms (application thread progress). Either can remove a member.
- 4
Common mistake: a regex subscription like orders-.* in a busy cluster. Every new matching topic causes a rebalance for the whole group.
- 5
Version note: cooperative rebalancing (2.4+) reduces the interruption but not the number of rebalances. Kafka 4.0 makes the KIP-848 protocol generally available, where the broker drives assignment and members no longer stop the world, and config names differ (for example, session and heartbeat timeouts become broker-controlled). Verify what your cluster and client versions use.
Scenario Questions
0-2 years experience
2-5 years experience
5-8 years experience
- 1A group uses a regex subscription and rebalances each time a team creates a new matching topic. How do you confirm that is the trigger and what do you propose instead?
- 2Explain JoinGroup and SyncGroup, the role of the group leader, and what generation fencing prevents. Why does a slow member make the whole rebalance slow?
8+ years experience
- 1Compare the classic group protocol with the KIP-848 protocol in terms of who computes assignment, what pauses, and what operational risks you would assess before migrating a critical group.
- 2You must choose session timeout, heartbeat interval, poll interval and static membership policy for a fleet running on spot instances with frequent restarts. Justify each setting and the failure modes you accept.