What is the difference between producer retries and idempotent production?

Question

Accepted Answer

Retries solve delivery: if a produce request fails with a transient error or times out, the producer sends it again. The problem is that a timeout is ambiguous. The broker may have written the batch and the acknowledgement was lost, so a blind retry writes the same records twice. Retries alone therefore give at-least-once delivery. Idempotent production solves duplication. The broker assigns the producer a producer ID, and the producer attaches a sequence number to each batch per partition. If the broker sees a sequence it has already written, it drops the batch and still acknowledges success. If the sequence jumps ahead, it rejects the batch with an out-of-order error. Retries plus idempotence give no duplicates and preserved order within a partition for the lifetime of that producer. import java.util.Properties;
import org.apache.kafka.clients.producer.ProducerConfig;

public class IdempotentConfig {
  public static Properties build() {
    Properties p = new Properties();
    p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    p.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
    p.put(ProducerConfig.ACKS_CONFIG, "all");                       // required by idempotence
    p.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"); // must be <= 5
    p.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, "120000");     // upper bound on retrying
    return p;
  }
} Scope of the guarantee: idempotence covers one producer instance (one producer ID) writing to one partition. If the application restarts it gets a new producer ID, so re-sending the same business event after a restart is not deduplicated. Cross-partition or cross-session atomicity needs transactions (transactional.id), which also fence zombie producers. That is a different tool with extra cost. Ordering: without idempotence, retries with max.in.flight.requests.per.connection above 1 can reorder records, because a failed batch can be retried after a later batch succeeded. Idempotence preserves order with up to 5 in flight. Trade-off: idempotence costs little (a little broker state and sequence tracking) and I enable it everywhere. The real cost is the constraints it imposes: acks=all and bounded in-flight requests. Common mistake: thinking retries=0 is the safe way to avoid duplicates. It trades duplicates for data loss on transient errors. The correct fix is idempotence, not disabling retries. Common mistake: assuming idempotence gives end-to-end exactly-once. Consumers can still process a record twice, so downstream handlers need their own deduplication. Version note: idempotence is enabled by default from client 3.0 provided you do not set conflicting configs such as acks=1. The retries default is effectively unbounded since 2.1, with delivery.timeout.ms as the real limit.

What is the difference between producer retries and idempotent production?

Retries re-send a request that may have failed; idempotence makes the broker discard the duplicates those retries can create

Scenario Questions

Follow-up Questions