What happens when a group has more consumers than partitions?

Question

Accepted Answer

A partition is assigned to at most one consumer in a group at a time. That rule is what preserves per-partition ordering. So if a topic has 4 partitions and the group has 6 consumers, four consumers each own one partition and two own nothing. The idle ones still join the group, send heartbeats and poll, but they fetch no data. Adding more consumers beyond the partition count gives no extra throughput, only extra cost and a slightly larger group to coordinate. Idle consumers are not useless. They act as hot standbys: if an active member dies, a rebalance gives its partitions to an idle member that is already running, which can shorten recovery compared with waiting for a new pod to start. The contrast with a classic queue is worth stating: in a competing-consumers queue, adding workers adds throughput up to the number of messages in flight. In Kafka the limit is structural, because the partition is the unit of assignment. If you need more parallelism, you increase partitions (which remaps keys and cannot be undone), make processing faster, or add concurrency inside each consumer while preserving key order. # Topic with 4 partitions, group with 6 members: 2 members show no partitions
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing --members --verbose

# Output shows each member's assigned partitions; idle members list none.
# Check how many partitions the topic has before scaling out:
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders Trade-off: over-provisioning consumers buys fast failover at the price of idle resources. Over-provisioning partitions buys scaling headroom at the price of broker overhead and key remapping on change. Multiple topics: with several subscribed topics, whether a consumer is idle depends on the assignor. RangeAssignor can leave the same members idle across topics, while others spread partitions more evenly, so check the actual assignment rather than assuming. Common mistake: scaling a lagging group by adding consumers past the partition count and expecting lag to fall. First compare consumers to partitions. Common mistake: autoscaling consumer pods on CPU or lag without capping replicas at the partition count. The autoscaler creates idle pods and wastes money. Common mistake: assuming idle consumers are harmless to the group. Each one is still a member, so its restarts and failures still trigger rebalances for everybody. Version note: share groups (KIP-932) are designed to let more consumers than partitions share work with queue-like semantics, but availability and maturity depend on your Kafka release, so verify before relying on it.

What happens when a group has more consumers than partitions?

The extra consumers sit idle because a partition can be owned by only one member of a group, so partition count is the ceiling on useful parallelism

Scenario Questions

Follow-up Questions