What problem does the transactional outbox pattern solve?

The transactional outbox pattern solves the dual-write problem: a service needs to update its database and publish an event to Kafka, but these are two separate systems with no shared transaction. If the service writes to the database and then publishes to Kafka, and the publish fails, the database is updated but the event is lost, so downstream consumers never learn about the change. If the service publishes to Kafka and then writes to the database, and the database write fails, the event is published but the database does not reflect it, so consumers act on a change that did not happen. In both cases, the system is inconsistent. The outbox pattern solves this by writing the event to an outbox table in the same database transaction as the business data. The event is not published to Kafka by the application; instead, a separate process (a relay or a CDC connector) reads the outbox table and publishes the events to Kafka. Because the outbox write and the business write are in the same transaction, they either both commit or both roll back. The relay then publishes the events at least once, and consumers must be idempotent. This guarantees that every committed business change eventually produces an event, without the risk of a partial write.

The mechanism of the outbox pattern has three parts: the outbox table, the transaction, and the relay. The outbox table has columns for the event ID, the aggregate ID, the event type, the payload, and a timestamp. The application writes the business data and the outbox row in a single database transaction. If the transaction commits, both are durable; if it rolls back, neither is. The relay reads the outbox table and publishes the events to Kafka. The relay can be implemented in two ways: polling (the application or a separate process polls the outbox table for unpublished rows and publishes them) or CDC (a tool like Debezium reads the database's transaction log and publishes the outbox changes to Kafka). CDC is generally preferred because it is more efficient and does not add polling load to the database, but polling is simpler and does not require a CDC tool. The trade-off is between latency and complexity. Polling adds latency because of the polling interval; CDC is near-real-time but requires Debezium or a similar tool and access to the database's transaction log. Another trade-off is between at-least-once and exactly-once: the outbox pattern gives at-least-once delivery, so consumers must be idempotent. Version note: the outbox pattern is a general pattern, not a Kafka feature. It is often implemented with Debezium for CDC and Kafka Connect for publishing. If you use a managed database, check whether it supports CDC; some managed databases do not expose the transaction log. Also note that the outbox table grows over time and needs a cleanup process to delete published rows.

A common mistake is to publish to Kafka inside the database transaction, which does not work because Kafka is not transactional with the database. Another mistake is to use the database's trigger to publish to Kafka, which reintroduces the dual-write problem because the trigger runs after the transaction and can fail independently. A third mistake is to assume that the outbox pattern gives exactly-once delivery; it gives at-least-once, so consumers must handle duplicates. The trade-off is between consistency and complexity. The outbox pattern adds a table, a relay, and idempotency requirements on consumers, but it eliminates the dual-write problem and guarantees that every committed change produces an event. For systems where event loss is unacceptable, this is the right trade-off. Version note: some frameworks and platforms provide outbox implementations, such as Debezium's outbox event router, which transforms outbox rows into properly formatted Kafka events. Check the version and configuration of the CDC tool when implementing the pattern.