Kafka brokers are healthy, but consumer lag, database load and retries grow continuously. What hypotheses would you test?

Healthy brokers point away from Kafka and towards the consumers and what they depend on. Growing lag, rising database load and growing retries together suggest a positive feedback loop: the database slows down, handlers take longer or time out, retries add more load to the same database, which slows it further, and lag grows because consumers are stuck waiting. The first move is to stop the amplification while I collect evidence, not to add capacity. The hypotheses I would test are, in order of likelihood: the database is saturated (connections, locks, slow queries, IOPS) for a reason that preceded the lag, such as a bad query plan, a missing index, a long transaction or a batch job; the consumer's concurrency or retry policy is overwhelming a database that cannot absorb it (immediate retries with no backoff, too many threads, a catch-up burst after a pause); connection pool exhaustion is turning slowness into timeouts and more retries; rebalances caused by exceeding max.poll.interval.ms are redelivering records, so the same work is done several times; a poison record or hot row is causing lock contention; or the idempotency check itself (a dedupe lookup per event) became the expensive query.

Each hypothesis has a discriminating test. Database side: active connections versus pool size, lock waits, slow query log, CPU and I/O saturation, and when each began relative to the first lag increase. Consumer side: per-attempt retry counts, handler latency histograms, pool wait time, poll-idle ratio, rebalance rate and duplicate rate. Little's law gives a sanity check: concurrency equals throughput times latency, so if the database latency went from 5 ms to 500 ms, the same thread count now demands 100 times more connections than the database can serve. Then I run controlled experiments: reduce consumer concurrency or pause partitions and see whether database latency recovers within seconds, which proves the consumers are part of the cause; switch retries to exponential backoff with jitter and a cap; add a circuit breaker so a failing dependency pauses consumption instead of being hammered. Pausing partitions while continuing to poll keeps the consumer in the group, avoiding extra rebalances. Only after the loop is broken do I address the root cause (the slow query, the missing index, capacity) and size concurrency to what the database can sustain.