Questions
1 of 5
Basics
Cluster Architecture
Topics, Partitions and Records
Producers
Consumers
Consumer Groups
Offset Management
Replication and Fault Tolerance
Storage Internals
Networking and Request Processing
Performance and Throughput
Delivery Semantics
Transactions
Kafka Streams
Kafka Connect
Schema and Serialization
Security
Error Handling and Retry Patterns
Ordering and Partitioning Design
Testing
Monitoring and Observability
Debugging and Troubleshooting
Deployment and Operations
Admin and CLI Operations
Reliability and Disaster Recovery
Integration Patterns
Modern Kafka Architecture
Advanced Internals
Scalability and Capacity Planning
Architecture and System Design
Production Best Practices
Expert Edge Cases
Versioning and Upgrades
Admin and CLI Operations
Senior End-to-End Scenarios
Difficulty: 6/10
Production architecture
A transactional outbox, a keyed and versioned event topic, a schema registry, an idempotent producer, and idempotent consumers with a retry and dead-letter path
I would start with the contract, not the cluster. The API's job is to accept the order and make a durable decision; Kafka's job is to tell other services what happened. The first design risk is the dual write: saving the order to the database and then publishing to Kafka are two writes with no shared transaction, so a crash between them leaves the database and the event stream disagreeing. I would remove that with a transactional outbox: the handler writes the order row and an outbox row in one local database transaction, and a separate relay (a poller or a CDC tool such as Debezium) publishes outbox rows to Kafka and marks them published. The relay is at-least-once, so the event carries a stable eventId created when the business event is created, never at send time.
Topic and contract design follows. One topic per event stream such as orders.events (not one topic per event type unless consumers really need to subscribe separately), keyed by orderId so every event for one order lands in one partition in order. Partition count is sized from measured peak throughput and expected consumer parallelism, with headroom, because changing it later remaps keys. Events are immutable facts with an envelope: eventId, eventType, schemaVersion, occurredAt, orderId, and a payload. I would put the schema in Avro or Protobuf behind a schema registry with backward-compatible evolution enforced in CI, and I would never publish internal database rows as the contract. Producers use acks=all, idempotence and a bounded delivery.timeout.ms on a topic with replication factor 3 and min.insync.replicas=2. Consumers own their side of the contract: process, then commit; deduplicate on eventId or use naturally idempotent writes; send permanently failing records to a dead-letter topic with error metadata and an owner and replay procedure; and use bounded retries so a transient fault does not block a partition forever.
javascript
- 1
Trade-off: outbox with a poller is simple but adds publish latency and database polling load. CDC (for example Debezium reading the outbox table) lowers latency and load but adds Kafka Connect and a connector to operate. I pick CDC when volume or latency requires it.
- 2
Trade-off: a single relay instance preserves per-order ordering simply but is a throughput and availability limit. Scaling the relay needs sharding by aggregate ID, or CDC, which preserves commit order per row.
- 3
Trade-off: thin events (IDs only) keep the contract small but force consumers to call the API back. Fat events (full state) decouple consumers but make the schema a larger, more carefully governed contract. For orders I usually publish enough state that common consumers need no callback.
- 4
Common mistake: calling producer.send() inside the request handler after the database commit and assuming it is safe. A crash or a Kafka timeout in that gap loses the event or forces the API to guess.
- 5
Common mistake: keying by something with skew or no business meaning (a random UUID per event, or a tenant ID alone). Key by the entity whose events must be ordered.
- 6
Common mistake: treating idempotent producers as end-to-end deduplication. They deduplicate retries within one producer session, not outbox re-publication after a relay restart, so consumers still dedupe on eventId.
- 7
Common mistake: no dead-letter or poison-message policy, so one malformed event blocks a partition and lag grows while everything looks healthy.
- 8
Version note: idempotence is the default from client 3.0, and KRaft is the only metadata mode from 4.0. The schema registry is not part of Apache Kafka itself; it comes from a vendor or compatible open implementation, so state which one you assume.
Scenario Questions
0-2 years experience
2-5 years experience
5-8 years experience
- 1Marketing wants a new consumer to rebuild its state from all historical orders. What must be true about retention, compaction and schemas, and how do you bootstrap it without affecting existing consumers?
- 2Compare a polling outbox relay with CDC on the outbox table for an API handling 3,000 orders per second, covering ordering, latency, failure modes and operations.
8+ years experience
- 1Ten teams will consume order events. Define the contract governance: ownership, naming, compatibility mode, deprecation policy, and how breaking changes are detected before deployment.
- 2Leadership asks whether orders should be event-sourced in Kafka instead of using a database plus outbox. Give your decision criteria, the risks (schema evolution, deletion rights, query needs) and your recommendation.