Questions
3 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 5/10
Partitioning and ordering
Size from target throughput and consumer parallelism, add growth headroom, and respect ordering and operational limits
I treat it as a sizing calculation with guardrails, because the count is hard to change later. First, estimate peak target throughput T for the topic. Measure, do not guess, the throughput one partition can sustain for producers (Pp) and for a consumer doing the real processing (Pc). The minimum is max(T/Pp, T/Pc). Example: T = 200 MB/s, Pp = 20 MB/s, Pc = 10 MB/s gives max(10, 20) = 20. Then I add headroom for growth, typically 1.5x to 3x, and round to a number with many divisors (for example 48) so consumers can be spread evenly. The consumer side usually dominates, because processing is slower than appending.
- 1
Consumer parallelism: the count caps useful consumers in a group, so I size for the maximum consumer count I expect to run, not today's count.
- 2
Ordering: if per-key order matters, the key space defines what is possible. Increasing partitions later changes key mapping, which is why I over-provision up front for keyed topics rather than expand later.
- 3
Operational cost: each partition adds file handles, replication load, producer batch memory, controller work and rebalance time. Too many partitions hurt end-to-end latency and recovery time.
- 4
Retention and storage: per-partition size matters for recovery and rebalancing; very large partitions take long to move.
- 5
Trade-off: over-provisioning wastes resources but is cheap to run; under-provisioning forces a painful repartition. I lean towards modest over-provisioning for important keyed topics, and towards starting small for stateless, unkeyed ones where growing is harmless.
- 6
Common mistake: copying a number from a blog post (100 everywhere) or from the consumer count of today. Another is expecting to fix a hot key by adding partitions, which it will not do.
- 7
Version note: older ZooKeeper-based guidance on partitions per broker is conservative for KRaft, and Kafka 4.0 is KRaft only. Validate limits with a load test on your version and hardware.
javascript
Scenario Questions
0-2 years experience
2-5 years experience
- 1A team measures 5 MB/s per consumer thread and expects 150 MB/s at peak. Calculate a reasonable partition count and explain your headroom choice.
- 2A topic was created with 6 partitions and traffic has now grown 10x. Compare adding partitions on the same topic with creating a new topic and migrating.
5-8 years experience
- 1A keyed topic feeds a stateful aggregation. Marketing predicts traffic will triple in a year. How do you size now so you avoid repartitioning, and what is the cost of being wrong either way?
- 2Your cluster hosts thousands of partitions per broker and controller failover and rolling restarts have become slow. How do you decide which topics to consolidate and how?
8+ years experience
- 1Design a self-service topic provisioning policy that proposes partition counts from declared throughput and consumers, and rejects unreasonable requests. What inputs, formulas and guardrails do you use?
- 2A latency-sensitive topic uses 500 partitions and end-to-end p99 is poor even though throughput is low. What mechanisms could be responsible and how would you test each?