Plan Kafka capacity for growth from 50 TB/day to 500 TB/day.

Planning for 10x growth from 50 TB/day to 500 TB/day requires modeling storage, replication, network, partitions, retention, recovery, and headroom. Start with the storage model: 500 TB/day of produce data. With replication factor 3, the raw storage per day is 1.5 PB. With 7-day retention, the total raw storage is 10.5 PB. Add 30% headroom for operational overhead, index files, and unexpected growth, giving about 13.65 PB. If compression is used, divide by the compression ratio; for example, a 3:1 compression ratio reduces 13.65 PB to 4.55 PB. The network model: 500 TB/day is about 5.8 GB/s average. Peak may be 2-3x the average, so plan for 15-20 GB/s. With replication factor 3, the replication traffic is 2x the produce traffic, so total network is about 3x the produce rate: 17.4 GB/s average, 50-60 GB/s peak. This requires a high-bandwidth network (25-100 Gbps per broker) and many brokers. The partition model: choose a partition count based on the target consumer parallelism and throughput. If each partition can handle 10 MB/s, 500 TB/day (5.8 GB/s) requires about 580 partitions for produce alone; with headroom and consumer parallelism, plan for 2,000-5,000 partitions. The broker model: a modern broker with NVMe SSDs can handle 100-200 MB/s of produce traffic per broker, depending on replication and acks. For 5.8 GB/s, you need 30-60 brokers for throughput; for 10.5 PB of storage, you need 100-200 brokers with 50-100 TB each. Storage is usually the limiting factor. Version note: tiered storage can offload older segments to object storage, reducing local storage requirements and allowing you to keep more data with less local disk. If you use tiered storage, the local storage is only the hot data (e.g., 1-2 days), and the remote storage is the long-term retention. This changes the broker count significantly.

The mechanism for the capacity plan has several parts. Storage: each broker has a disk capacity; the total storage is the sum of all broker disks. With replication factor 3, each record is stored 3 times, so the effective storage is 1/3 of the raw storage. Plan for the raw storage plus headroom. Network: each broker has a NIC; the total network is the sum of all NICs. The network must handle produce, replication, and consume traffic. Replication is the largest component because each record is sent from the leader to two followers. Consumer traffic depends on the number of consumer groups. Partitions: each partition has overhead; plan for the target partition count and ensure that the brokers can handle it. Recovery: after a broker failure, the remaining brokers must handle the load and re-replicate the lost partitions. The cluster must have enough headroom to survive a broker failure without saturation. The trade-off is between cost and resilience. More brokers and more disk give more headroom and faster recovery but cost more. Fewer brokers are cheaper but risk saturation during failures or peaks. A good practice is to plan for N+2 broker failures: the cluster can lose two brokers and still have enough capacity. Version note: cloud-based Kafka services often provide auto-scaling and tiered storage, which can simplify capacity planning. Self-managed clusters require more careful planning. Always test with a realistic workload and monitor actual usage to refine the plan.

A common mistake is to plan for average throughput instead of peak. If the peak is 3x the average, the cluster will be overwhelmed during peak. Another mistake is to ignore replication in the storage and network estimates. A third mistake is to under-provision recovery headroom; a cluster with exactly the capacity for the workload will be saturated when a broker fails. The trade-off is between over-provisioning and under-provisioning. Over-provisioning costs more but gives resilience and room for growth. Under-provisioning saves money but risks outages. A good practice is to plan for peak plus 30-50% headroom, and to monitor utilization to know when to add brokers. For 10x growth, the plan should be phased: add brokers in stages, monitor, and adjust. Version note: KRaft supports larger clusters with faster metadata operations, which is important at 500 TB/day. Tiered storage can reduce local storage requirements, but it adds remote read latency and requires network bandwidth to the object store. Consider the cost of the object store and the network egress when designing tiered storage.