How would you design a high-availability, horizontally scalable Go backend service?

Question

Accepted Answer

Stateless: all session state, cache, and persistent data lives in Postgres, Redis, or object storage Connection pooling: pgxpool with MaxConns = (postgres_max_connections / replicas) - buffer Health checks: /readiness (DB connected, dependencies available) and /liveness (process alive) for Kubernetes Graceful shutdown: drain in-flight requests before exiting on SIGTERM Zero-downtime deploy: rolling updates work because the service is stateless and backward-compatible import "github.com/sony/gobreaker"

// Circuit breaker for downstream service calls
cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "payment-service",
    MaxRequests: 3,              // half-open state: allow 3 requests to test recovery
    Interval:    10 * time.Second,
    Timeout:     60 * time.Second, // open state duration
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
        return counts.Requests >= 10 && failureRatio >= 0.6
    },
    OnStateChange: func(name string, from, to gobreaker.State) {
        log.Printf("circuit %s: %s -> %s", name, from, to)
    },
})

func callPaymentService(ctx context.Context, req *PaymentRequest) (*PaymentResponse, error) {
    result, err := cb.Execute(func() (interface{}, error) {
        return paymentClient.Process(ctx, req)
    })
    if err != nil {
        if err == gobreaker.ErrOpenState {
            return nil, errors.New("payment service unavailable") // degrade gracefully
        }
        return nil, err
    }
    return result.(*PaymentResponse), nil
} Horizontal scaling: deploy more replicas — works because service is stateless Distributed locking: Postgres advisory locks or Redlock for critical sections across replicas Caching: Redis for session data, hot DB queries, and rate limit counters Bulkhead: separate goroutine pools for different request types to prevent cascade failures Retry with exponential backoff and jitter for transient failures in downstream calls

How would you design a high-availability, horizontally scalable Go backend service?

Build a stateless service with all state in external stores, deploy N replicas behind a load balancer, use circuit breakers for downstream dependencies, and design for graceful degradation.

How would you design a high-availability, horizontally scalable Go backend service?

Build a stateless service with all state in external stores, deploy N replicas behind a load balancer, use circuit breakers for downstream dependencies, and design for graceful degradation.

Scenario Questions

Follow-up Questions