Questions
4 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 8/10
Partitioning and ordering
Confirm it is data skew, find whether it is a hot key or a bad key distribution, then spread load at the finest ordering granularity the business allows
I would first confirm the symptom with numbers, then classify the cause, because the fixes are different. To measure, I compare log end offsets and on-disk size per partition and look at per-partition consumer lag. A big partition on one broker can also be a broker-level problem (leader imbalance), so I check leader distribution too. Then I sample keys landing in the hot partition: one dominant key is a hot key; a few keys dominating is low key cardinality; many keys hashing unevenly is rare with murmur2 and usually points to a key format problem or a custom partitioner bug; null keys with an explicit partition set by code is another common culprit.
javascript
The fix depends on what ordering the business really requires. Option one: use a finer key, for example tenantId plus userId instead of tenantId, keeping per-entity ordering while spreading load. Option two: salt only the hot keys, so a hot key is spread over N partitions and downstream aggregation recombines results; this gives up ordering for that key across the sub-streams. Option three: a custom partitioner that spreads known hot keys and hashes everything else normally. Option four: if ordering does not matter, drop the key and let the sticky partitioner balance. Adding partitions does not fix a hot key, since the key still maps to one partition, and it breaks key mapping for everyone.
javascript
- 1
Trade-off: every technique that spreads a hot key gives up strict ordering for that key. State that explicitly and ask whether the real requirement is ordering per user, per session, or per aggregate.
- 2
Consumer-side mitigation: if one partition's consumer is the bottleneck, process records from that partition concurrently with a key-aware worker pool, preserving order per key inside the consumer. This is a workaround, not a cure for storage and broker skew.
- 3
Common mistake: assuming skew means the hash function is bad. murmur2 is well distributed, so the cause is almost always the key choice, not the hash.
- 4
Common mistake: adding partitions or consumers as the first reaction. Neither moves traffic off a hot key.
- 5
Every producer must use the same custom partitioner, or ordering and locality assumptions silently break. Plan the rollout and monitor partition balance after the change.
- 6
Version note: Cruise Control and tooling for leader and replica balancing exist for broker-level skew, but their availability depends on your distribution or managed offering.
Scenario Questions
0-2 years experience
2-5 years experience
- 1You find that 70 percent of a topic's records have the key 'unknown' because of a bug upstream. How do you confirm it from Kafka alone, and how do you fix it without losing data?
- 2One broker has noticeably higher CPU and disk I/O than the others but partition sizes look even. What else could create imbalance and how do you check?
5-8 years experience
- 1A SaaS platform keys by tenantId and one enterprise tenant generates 40 percent of traffic. Propose two designs that spread the load and compare what each costs in ordering and complexity.
- 2After you introduce key salting, downstream counts are occasionally wrong during deploys. What failure modes would you investigate?
8+ years experience
- 1Skew is recurring across dozens of topics owned by different teams. What platform-level detection, guidance and automation would you build instead of fixing topics one at a time?
- 2A regulated workflow needs per-tenant ordering, and the biggest tenant exceeds what one partition and one consumer can process. What are your options, and what would you tell the business about the trade-offs?