When should you add brokers versus increasing application concurrency?

The decision between adding brokers and increasing application concurrency depends on where the bottleneck is. If the bottleneck is broker-side—disk I/O, network bandwidth, CPU, or replication—adding brokers increases the cluster's capacity to handle more partitions, more replication, and more produce/fetch requests. If the bottleneck is application-side—the consumer is slow because of expensive processing, synchronous downstream calls, or limited CPU on the consumer instances—adding brokers will not help. You need to increase application concurrency by adding consumer instances (if partitions are available), optimizing the processing, or parallelizing the work. The first step is to identify the bottleneck with metrics: check broker disk, network, and request latency, and check consumer processing time, CPU, and downstream latency. If the broker is saturated but the consumer is idle, add brokers. If the consumer is saturated but the broker is idle, increase consumer concurrency. If both are saturated, you may need to do both. The trade-off is between cost and complexity: adding brokers increases infrastructure cost and operational complexity; increasing consumer concurrency increases application complexity and may require more partitions.

The mechanism for each option is different. Adding brokers increases the cluster's total disk, network, and CPU capacity. It allows more partitions to be distributed across more brokers, reducing the load per broker. It also increases the replication capacity, which improves durability and reduces under-replicated partitions. Increasing application concurrency increases the number of consumers processing records in parallel. For a consumer group, the maximum concurrency is the number of partitions. If you have 10 partitions and 10 consumers, adding more consumers will not help; you need more partitions. If you have 10 partitions and 5 consumers, adding 5 more consumers will double the parallelism. Alternatively, you can increase concurrency within a consumer by processing records in parallel (e.g., a thread pool), but this breaks ordering unless you handle it carefully. The trade-off is between horizontal scaling (more consumers) and vertical scaling (more threads per consumer). Horizontal scaling is simpler for ordering but limited by partitions; vertical scaling can exceed the partition limit but requires careful ordering management. Version note: Kafka's consumer group protocol has improved with cooperative rebalancing and static membership, which make it easier to scale consumers without disrupting processing. If you are on an older version, rebalances may be more disruptive, so adding brokers (which does not trigger consumer rebalances) may be preferable to adding consumers.

A common mistake is to add consumers when the bottleneck is the broker, which increases load on the broker without improving throughput. Another mistake is to add brokers when the bottleneck is the consumer, which wastes money without improving throughput. A third mistake is to ignore the partition count: if the topic has 10 partitions and you already have 10 consumers, adding more consumers will not help. The trade-off is between infrastructure cost and application complexity. Adding brokers is a straightforward infrastructure change but costs more; increasing consumer concurrency requires application changes and may be limited by partitions. A good approach is to monitor both broker and consumer metrics continuously and to scale the layer that is the bottleneck. Use auto-scaling for consumers where possible, and plan broker capacity with headroom for peaks. Version note: cloud-based Kafka services often make it easy to add brokers, but the cost can be high. Self-managed clusters require more planning. In both cases, the decision should be data-driven, not based on assumptions.