A producer times out but the record may already exist in Kafka. How should the application reason about retrying?

The core fact is that a timeout means 'unknown', not 'failed'. The request may have been written and replicated, with only the response lost. So the application must never assume the record is absent. I reason in two layers. Layer one is the producer's internal retries: with enable.idempotence=true, acks=all and a sensible delivery.timeout.ms, the producer retries the same batch with the same sequence number, and the broker deduplicates, so these retries are safe and I do not interfere. Layer two is what happens when the producer finally gives up and the send callback or future reports an error. Now the outcome is still unknown. If the application calls send() again with the same payload, that is a brand new batch with a new sequence number, and the broker cannot deduplicate it. If the first one actually landed, you now have a duplicate.

So my approach is: first, tune delivery.timeout.ms and request.timeout.ms so transient problems are absorbed by internal idempotent retries and the application sees an error only when the cluster has been unhealthy for a long time. Second, treat an application-level resend as at-least-once and make it safe by attaching a stable event ID to every record (generated before the first send and reused on resend) so consumers can deduplicate. Third, for the source-of-truth side, avoid sending directly from request handlers where a lost ack triggers ad hoc retry logic. Write the event to a database outbox in the same transaction as the state change and publish from there, so the retry loop is persistent and replays the same event ID. If the unit of work is read-process-write across Kafka topics, use transactions, which make the write and the offset commit atomic.