Design an event-driven order system using Kafka, relational databases, outbox and projections.

An event-driven order system has three main components: the order service (write model), the event backbone (Kafka), and the read models (projections). The order service handles commands like CreateOrder, ConfirmOrder, and ShipOrder. It validates the command, applies business rules, and writes the order state to its relational database. In the same transaction, it writes a domain event to an outbox table. A CDC connector (Debezium) reads the outbox table and publishes the events to Kafka. The events are keyed by order ID, so all events for an order go to the same partition and are ordered. The read models consume the events and update projections: an order view for customer queries, a fulfillment view for the warehouse, and an analytics view for reporting. Each projection is a separate consumer group with its own offsets and its own read store. The system is eventually consistent: the read models lag behind the write model by the replication and processing time, but they can be scaled independently and optimized for their query patterns. The trade-off is between consistency and scalability. The order service is the source of truth; the projections are derived and can be rebuilt by replaying Kafka.

The mechanism of the system has several layers. At the write layer, the order service uses the outbox pattern to ensure that every committed order change produces an event. The outbox table is in the same database as the order data, so the transaction is local and fast. At the event layer, Kafka stores the events in order per order ID and retains them for replay. The events are schema-managed with a schema registry, and the compatibility policy is FULL_TRANSITIVE to protect consumers. At the read layer, each projection is a Kafka consumer that applies events to its read store. The projections must be idempotent because Kafka delivers at least once. They use the event ID or the aggregate version to deduplicate, and they use upserts rather than inserts. At the operational layer, the system monitors consumer lag for each projection, CDC lag for the outbox, and the error rate. If a projection has a bug, it can be rebuilt by resetting its consumer group offset to the beginning and replaying the events. If the outbox relay fails, the order service continues to accept commands, and the events are published when the relay recovers. The trade-off is between the number of projections and the operational cost. Each projection adds a consumer group, a read store, and monitoring; but each projection can be optimized for a specific query pattern, which is the main benefit. Version note: Debezium for CDC, Kafka Connect for the relay, and Kafka Streams for projections are the standard tools. If you use a managed Kafka service, check whether it supports Connect and CDC connectors.

A common mistake is to publish events directly from the order service instead of using the outbox, which reintroduces the dual-write problem. Another mistake is to make the projections non-idempotent, which causes duplicates when events are re-delivered. A third mistake is to ignore schema evolution; if the order event schema changes incompatibly, the projections break. The trade-off is between consistency and availability. The order service is strongly consistent for its own data (the database transaction), but the read models are eventually consistent. The business must accept that a query immediately after a command may return stale data. For an order system, this is usually acceptable: the customer sees the order confirmation from the write model, and the read models catch up within milliseconds to seconds. The system must also handle failures: if the CDC connector fails, the outbox grows but the order service continues; if a projection fails, its lag grows but the other projections continue. This isolation is a key benefit of the event-driven design. Version note: the outbox pattern with Debezium's EventRouter is the standard way to publish domain events from a relational database. Kafka Streams is a common choice for projections because it provides state stores and exactly-once processing. If you need interactive queries, Kafka Streams can serve the read model directly.