When should a record go to a dead-letter topic instead of being retried indefinitely?

A record should go to a dead-letter topic (DLT) when retrying it will not succeed, or when the cost of retrying it indefinitely exceeds the value of eventually processing it. The decision is based on failure classification. Transient failures are those that are likely to resolve on their own: network timeouts, temporary unavailability of a downstream service, rate limiting, or database connection pool exhaustion. These should be retried with backoff. Permanent failures are those that are tied to the content of the record and will never succeed with the same code: deserialization errors, schema violations, missing required fields, foreign key violations, or business rule violations. These should go to the DLT immediately or after a small number of retries to confirm the failure is deterministic. There is also a third category: failures that are ambiguous, where you cannot tell whether the issue is transient or permanent. For these, a bounded number of retries with backoff is the right approach, followed by the DLT if the retries are exhausted.

The mechanism for deciding is to define a retry policy per exception type. For example, a TimeoutException or a 503 from an HTTP call is retryable; a JsonParseException or a ConstraintViolationException is not. You also need a maximum retry count or a maximum age for the record. Even for transient failures, retrying forever is dangerous because it can block the partition and hide the fact that the downstream service has been down for hours. A common pattern is: 3-5 retries with exponential backoff over a few minutes, then the DLT. The DLT record should carry enough context to diagnose and replay: the original topic, partition, offset, the exception class and message, the consumer group, and the timestamp of the first failure. Without this metadata, the DLT is not actionable. The trade-off is between completeness and progress. Sending a record to the DLT means it is not processed by the main flow, which may be unacceptable for some records. But retrying forever means the entire partition is blocked, which is worse. The DLT is a controlled way to remove a record from the main flow while preserving it for later analysis and replay.

A common mistake is to send records to the DLT without monitoring it. A DLT that nobody looks at is a silent data loss mechanism. You need alerts on DLT volume, and a process for triaging and replaying DLT records. Another mistake is to send records to the DLT for transient failures after only one attempt, which discards records that would have succeeded on a second try. A third mistake is to use the DLT as a permanent archive; it should have a retention policy and a replay path. The DLT is not a substitute for fixing the root cause. If records are consistently going to the DLT, you have a bug or a schema mismatch that needs to be fixed, not just quarantined. Version note: Kafka does not have a built-in DLT; the pattern is implemented by the consumer. Spring Kafka provides a DeadLetterPublishingRecoverer and a @DltHandler annotation. Kafka Streams has a similar concept with the 'continue on error' pattern, though it is less mature. Always check the framework version and its DLT support.