Questions
1 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 2/10
Commit and replay
An offset is a record's sequential position in a partition, and a consumer's committed offset is the next position its group will read after a restart
Every record appended to a partition gets a sequential, immutable offset (0, 1, 2, ...). Offsets are meaningful only within one partition, so a record's real identity is topic, partition, offset. A consumer's progress is tracked per partition in two distinct ways. The position is the offset of the next record the consumer will fetch, held in memory by the client and advanced by each poll(). The committed offset is the last position the group has durably saved to the internal __consumer_offsets topic, keyed by group, topic and partition. After a restart or rebalance, a consumer resumes from the committed offset, not from the in-memory position.
The detail people get wrong is the off-by-one: the committed offset means 'the next record to read', so after processing the record at offset 41 you commit 42. If you commit 41, the record at 41 is delivered again after a restart. Also, the committed offset belongs to the group, not to the consumer instance, which is why a replacement member continues exactly where a crashed one left off, and why different groups can read the same partition at completely different positions. The broker does not delete records when offsets are committed; retention is independent of consumption.
javascript
- 1
Trade-off: committing more often reduces duplicate reprocessing after a crash but adds commit requests and latency. Committing less often is cheaper but replays more records after a failure.
- 2
Common mistake: committing record.offset() instead of record.offset() + 1. The no-argument commitSync() commits the current position, which is already correct, so the off-by-one appears only when you build the offset map yourself.
- 3
Common mistake: confusing auto.offset.reset with committing. It applies only when the group has no committed offset or the committed offset is out of range, not on every restart.
- 4
Common mistake: thinking committed offsets live forever. For a group with no active members, committed offsets expire after offsets.retention.minutes (7 days by default in recent versions), and the group then restarts from auto.offset.reset. A long weekend is fine; a two-week pause may not be.
- 5
Common mistake: believing the offset says how many records exist. Offsets can have gaps (compaction, transaction markers), so subtracting offsets approximates but does not equal a record count.
- 6
Version note: offsets are stored in __consumer_offsets (since 0.9), not in ZooKeeper. Defaults such as the offset retention period have changed over releases, so verify them for your version.
Scenario Questions
0-2 years experience
2-5 years experience
- 1A new deployment of a consumer starts from the very beginning of the topic instead of where the old one stopped. Name three possible causes you would check.
- 2Your consumer group was stopped for ten days and, when restarted, reprocessed data from the latest offset only, skipping everything produced meanwhile. What mechanism explains this and how do you prevent it?
5-8 years experience
- 1Lag shows as zero but the business says some events were never processed. Walk through how committed offsets, positions and your commit logic could produce that picture.
- 2Compute consumer lag for a group from raw broker data. Which two numbers do you need per partition, and what situations make 'lag in records' misleading (compaction, transactions, gaps)?
8+ years experience
- 1Design the monitoring for committed offsets across 300 groups so you detect stalled, expired and rewound groups, not only high lag. What metrics, alerts and ownership model do you use?
- 2Argue for or against storing offsets outside Kafka (for example in the application database). What guarantees does each choice give, and what does each cost operationally?