Q42. What does exactly-once semantics mean in Kafka, and what does it not guarantee?

Exactly-once semantics (EOS) in Kafka means that a consume-transform-produce pipeline can read from a source topic, process the record, and write to a sink topic such that the sink sees each input record exactly once, even in the presence of failures. This is implemented via two mechanisms: the idempotent producer and transactions. The idempotent producer assigns a producer ID (PID) and a sequence number to each record, so the broker can deduplicate retries within a single producer session. Transactions extend this across multiple partitions and across the consume-transform-produce boundary by using a transaction coordinator and a two-phase commit protocol. The consumer uses isolation.level=read_committed to avoid seeing uncommitted or aborted records. This is a significant advancement and is what people mean when they say Kafka supports exactly-once.

What EOS does not guarantee is just as important. First, it does not make arbitrary external side effects exactly-once. If your processing writes to a database, calls an HTTP API, or sends an email, Kafka's transaction cannot roll back that side effect. The transaction only covers Kafka reads and writes. This is the single most common misconception. Second, EOS does not cover producer-to-Kafka writes outside of a transaction. If you produce without a transaction, the idempotent producer prevents duplicates on retry within a session, but if the producer crashes and restarts with a new PID, duplicates can still occur. Third, EOS does not cover consumer offset commits outside the transaction. If you commit offsets separately from your transactional writes, you can still get duplicates or loss. Fourth, EOS is scoped to the Kafka cluster; if you produce to a different cluster or a different system, the transaction does not extend there.

The trade-off is performance and complexity. Transactions add latency because of the two-phase commit and the transaction coordinator. They also require careful configuration: transactional.id must be unique and stable per producer instance, and consumers must use read_committed. In practice, EOS is most valuable for stream processing pipelines where the source and sink are both Kafka, such as Kafka Streams or a Flink job writing back to Kafka. For external sinks, the standard pattern is idempotent writes plus an outbox or inbox table, which gives you effectively-once behavior without Kafka transactions. Another version-dependent note: EOS was introduced in Kafka 0.11 and has been improved since. KIP-447 improved scalability of transactions in Kafka 2.5 by allowing a single producer to write to multiple topic partitions with better resource management. Always check the broker and client versions when relying on EOS features.