Why does careful offset committing not by itself provide exactly-once business effects?

An offset commit is a write to Kafka. A business effect, such as a database update, a payment call, an email or an HTTP request, is a write to some other system. There is no atomic operation spanning both. So whichever order you choose, a crash in the gap breaks something. Effect first, then commit: the effect happened, the commit did not, so the record is redelivered and the effect repeats (at-least-once, duplicates). Commit first, then effect: the commit happened, the effect did not, so the effect is lost (at-most-once). Careful commit ordering therefore only chooses which failure you get; it cannot eliminate both. And duplicates arise even without crashes: rebalances can hand a partition to a new consumer while the old one is still finishing work, a commit can fail after processing succeeded, and retries inside the handler can repeat a non-idempotent call.

What Kafka's exactly-once features cover is narrower than the phrase suggests. Idempotent producers deduplicate retries to a partition. Transactions with sendOffsetsToTransaction make 'write output records to Kafka and commit the input offsets' atomic, and read_committed consumers see only committed output, so a Kafka-to-Kafka consume-transform-produce pipeline (including Kafka Streams with exactly_once_v2) can be effectively exactly-once. The guarantee ends at Kafka's boundary: an external database, API or email is outside the transaction. For those, I get exactly-once business effects by making the effect idempotent (dedupe on a stable event ID, upsert by natural key, or pass an idempotency key to the external API), or by storing the consumer offset in the same local transaction as the effect, so both succeed or fail together, and seeking to the stored offset on assignment. Kafka does not support two-phase commit across arbitrary systems.