Questions
3 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 5/10
Polling and processing
Fetch size, broker wait time, per-poll batch size and the consumer's own processing capacity together set the throughput and latency trade-off
Think of the fetch path as a pipeline with two separate dials. The first is how the broker answers a fetch: fetch.min.bytes (default 1) is how much data the broker should accumulate before replying, and fetch.max.wait.ms (default 500) is the longest it will wait to reach that amount. With the defaults, the broker replies as soon as any data exists, which is low latency but small, frequent responses. Raising fetch.min.bytes makes responses larger and cheaper per byte, improving throughput and reducing broker and network load, but on a quiet topic a record can wait up to fetch.max.wait.ms before delivery.
The second dial is how much a response may contain: max.partition.fetch.bytes (default 1 MB) caps data per partition and fetch.max.bytes (default about 50 MB) caps the whole response. Both are soft limits, since a single oversized batch is still returned so the consumer cannot get stuck. Separately, max.poll.records (default 500) controls how many already-fetched records one poll() hands to the application, and does not change how much is fetched from the network. The senior insight is that all of these only matter if the consumer is fetch-bound. Most real consumers are processing-bound: if your handler takes 20 ms per record, no fetch tuning will raise throughput beyond 50 records per second per consumer thread. Little's law applies: required concurrency equals arrival rate times processing time.
javascript
- 1
Trade-off: a latency-sensitive consumer (alerts, user-facing) keeps small waits and small batches. A bulk consumer (ETL, analytics) raises fetch.min.bytes and fetch size to cut requests. I tune from metrics, not from rules of thumb.
- 2
Trade-off: larger fetch sizes mean more memory per consumer, roughly partitions assigned times max.partition.fetch.bytes in the worst case, so a consumer with many partitions can need a lot of heap.
- 3
Common mistake: raising max.poll.records and expecting more network throughput. It only changes batch size handed to the app, and can push you toward the max.poll.interval.ms limit.
- 4
Common mistake: setting fetch.min.bytes high on a low-traffic topic and then being surprised by half-second latency spikes, because the broker waits for fetch.max.wait.ms.
- 5
Common mistake: tuning fetch parameters when the real bottleneck is processing time or too few partitions. Check poll-idle-ratio-avg: if it is near zero the consumer is busy processing, not waiting on fetches.
- 6
Cost lever: follower fetching (client.rack with a rack-aware replica selector on the brokers, KIP-392, since 2.4) lets consumers read from a same-zone replica and cut cross-AZ network charges, at the price of slightly higher replication-lag-bound latency.
- 7
Version note: KIP-74 made the fetch size limits soft in 0.10.1; older advice treating them as hard caps is outdated. Also check defaults on your version, since some fetch-related defaults have been revisited in newer releases.
Scenario Questions
0-2 years experience
2-5 years experience
- 1A nightly ETL consumer reads 400 MB/s from the topic backlog but only reaches 60 MB/s. Which fetch settings would you check first and which metrics confirm it?
- 2A consumer assigned 200 partitions runs out of heap after you increase max.partition.fetch.bytes. Explain the mechanism and how you would size it.
5-8 years experience
- 1Alert-processing p99 latency is 600 ms although the handler takes 5 ms. Fetch settings were tuned for throughput last quarter. Diagnose and propose a profile.
- 2Cross-AZ data transfer is a large part of the cloud bill. Explain how consumers can reduce it, what the brokers must support, and what you give up.
8+ years experience
- 1Design a tiered consumer-configuration standard (real-time, standard, bulk) with justified fetch and poll settings, and describe how you would verify each tier in production.
- 2A consumer has high poll-idle-ratio and low CPU yet lag keeps growing during peaks. Argue why fetch tuning is the wrong lever and what the real constraints might be.