Questions
1 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 3/10
At-least-once, Exactly-once, Idempotency
At-Most-Once vs At-Least-Once: The Failure Points That Cause Loss or Duplication
Delivery semantics describe what happens to a message when something fails. At-most-once means a message is delivered zero or one time, never more. At-least-once means a message is delivered one or more times, never zero. The difference is entirely about the order of operations between processing and committing the offset. In Kafka, the consumer commits its position in the partition log to the __consumer_offsets topic. If you commit the offset before processing the message, and processing fails, the message is lost because the consumer will resume after it on restart. That is at-most-once. If you process the message first and then commit, and the process crashes after processing but before committing, the message will be re-delivered on restart. That is at-least-once.
The mechanism matters because it determines where duplication or loss can occur. With auto-commit enabled (enable.auto.commit=true), the consumer commits offsets periodically in the background, independent of whether your processing succeeded. This is effectively at-most-once if your processing is asynchronous or slow, because the commit can happen before processing finishes. To get at-least-once, you disable auto-commit (enable.auto.commit=false) and commit manually after processing. But at-least-once has a subtle failure point: if the consumer processes a batch, commits, then crashes before the next poll, no duplication occurs. If it processes, crashes before committing, the whole batch is re-delivered, including messages that were already processed. This is why at-least-once requires idempotent processing on the consumer side; otherwise you get duplicate side effects.
A common mistake is to think that at-least-once is always the safe choice. It is safer against loss, but it pushes the duplication problem onto the consumer. Another mistake is to assume that committing more frequently reduces duplication. It does not eliminate it; it only reduces the window. The trade-off is between commit frequency and throughput: committing every message is expensive and kills throughput, so most consumers commit per batch or per poll. On the producer side, at-most-once is achieved by setting acks=0 or acks=1 with no retries; if the broker does not receive the message, the producer does not know and does not retry. At-least-once on the producer side is achieved with acks=all and retries > 0; if the ack is lost, the producer retries, which can cause duplicates. Exactly-once producer semantics require idempotence and transactions, which is a separate topic.
javascript
- 1
At-most-once: commit offset before processing; failure causes loss, never duplication.
- 2
At-least-once: process then commit; failure causes duplication, never loss.
- 3
Auto-commit is effectively at-most-once if processing is asynchronous or slow.
- 4
Manual commit after processing gives at-least-once, but duplicates occur for the whole batch on crash.
- 5
Producer at-most-once uses acks=0 and no retries; producer at-least-once uses acks=all with retries.
- 6
At-least-once requires idempotent consumer processing to avoid duplicate side effects.
- 7
Exactly-once requires idempotent producer plus transactions; it is not the default.
Scenario Questions
0-2 years experience
2-5 years experience
5-8 years experience
- 1You are designing a payment processing pipeline. Which delivery semantic do you choose at each stage, and how do you handle the failure modes?
- 2A consumer group is reprocessing a large batch after a crash, causing duplicate downstream writes. How do you reduce the batch size and commit frequency trade-off?
8+ years experience
- 1You are migrating a legacy system to Kafka and need to choose delivery semantics for a pipeline that updates a ledger. How do you model the cost of duplication versus loss, and what compensating controls do you put in place?
- 2A regulator requires that no financial event be lost and that duplicates be detectable. How do you design the producer and consumer to meet both requirements simultaneously?