Describe how you would approach a Go service that needs to process Kafka events with exactly-once semantics.

Question

Accepted Answer

// Idempotent consumer: store processed event IDs
func (h *OrderEventHandler) Handle(ctx context.Context, msg *kafka.Message) error {
    var event OrderCreatedEvent
    if err := json.Unmarshal(msg.Value, &event); err != nil {
        return fmt.Errorf("unmarshal: %w", err)
    }

// Atomic: write domain state + mark event processed in one transaction
    return h.db.WithTransaction(ctx, func(tx *sql.Tx) error {
        // Check idempotency key — UNIQUE constraint prevents duplicates
        _, err := tx.ExecContext(ctx,
            `INSERT INTO processed_events (event_id, processed_at)
             VALUES ($1, NOW())
             ON CONFLICT (event_id) DO NOTHING`,
            event.ID,
        )
        if err != nil { return err }

// Process domain logic
        if err := h.orderRepo.Create(ctx, tx, &event.Order); err != nil {
            return err
        }
        return nil
    })
}

// Transactional outbox: produce events reliably
func (s *OrderService) Create(ctx context.Context, order *Order) error {
    return s.db.WithTransaction(ctx, func(tx *sql.Tx) error {
        // Write domain state
        if err := s.orderRepo.Create(ctx, tx, order); err != nil {
            return err
        }
        // Write outbox entry in SAME transaction
        _, err := tx.ExecContext(ctx,
            `INSERT INTO outbox (aggregate_id, event_type, payload)
             VALUES ($1, $2, $3)`,
            order.ID, "order.created", mustMarshal(order),
        )
        return err
    })
    // Separate outbox relay goroutine reads outbox and publishes to Kafka
} Idempotent consumer: processed_events table with UNIQUE(event_id) — ON CONFLICT DO NOTHING skips duplicates Transactional outbox: write event and state change in same DB transaction — relay publishes to Kafka separately Commit offsets only after successful DB write — ensures re-processing on failure Kafka transactions + transactional producers for Kafka-to-Kafka pipelines (not DB sinks) segmentio/kafka-go or confluent-kafka-go for production Kafka consumers in Go

Describe how you would approach a Go service that needs to process Kafka events with exactly-once semantics.

True exactly-once requires idempotent consumers with deduplication stored in the DB, the transactional outbox pattern for producing events, and Kafka consumer group offset commits tied to successful processing.

Describe how you would approach a Go service that needs to process Kafka events with exactly-once semantics.

True exactly-once requires idempotent consumers with deduplication stored in the DB, the transactional outbox pattern for producing events, and Kafka consumer group offset commits tied to successful processing.

Scenario Questions

Follow-up Questions