How would you design a high-availability, horizontally scalable Go backend service?
Build a stateless service with all state in external stores, deploy N replicas behind a load balancer, use circuit breakers for downstream dependencies, and design for graceful degradation.
- 1
Stateless: all session state, cache, and persistent data lives in Postgres, Redis, or object storage
- 2
Connection pooling: pgxpool with MaxConns = (postgres_max_connections / replicas) - buffer
- 3
Health checks: /readiness (DB connected, dependencies available) and /liveness (process alive) for Kubernetes
- 4
Graceful shutdown: drain in-flight requests before exiting on SIGTERM
- 5
Zero-downtime deploy: rolling updates work because the service is stateless and backward-compatible
- 1
Horizontal scaling: deploy more replicas — works because service is stateless
- 2
Distributed locking: Postgres advisory locks or Redlock for critical sections across replicas
- 3
Caching: Redis for session data, hot DB queries, and rate limit counters
- 4
Bulkhead: separate goroutine pools for different request types to prevent cascade failures
- 5
Retry with exponential backoff and jitter for transient failures in downstream calls
Scenario Questions
0-2 years experience
- 1We have a single Go instance behind a load balancer. If that instance crashes, what steps would you add to keep the API available?
- 2How would you set up a basic health‑check endpoint and configure the load balancer to route traffic only to healthy Go pods?
- 3You need to add a new endpoint that must stay up during deployments. Describe how you’d structure the code and deployment to avoid downtime.
2-5 years experience
- 1After scaling the service to five instances, you notice latency spikes under load. Walk me through how you’d debug the issue and what design changes you might consider.
- 2During a rolling deployment one instance starts returning 500 errors. How would you detect, isolate, and fix the problem without affecting users?
- 3Explain the trade‑offs between using a single shared database versus sharding data across instances for a highly available Go service.
5-8 years experience
- 1Design a high‑availability architecture for a Go backend that must handle 10k QPS, survive an entire zone outage, and support zero‑downtime deployments. Discuss components, data consistency, and failure detection.
- 2Your service relies on in‑memory caches that aren’t replicated. How would you redesign caching to eliminate a single point of failure while preserving performance?
- 3What strategies would you use to ensure graceful degradation when downstream services become unavailable, and how would you implement them in Go?
8+ years experience
- 1Our organization is moving from a monolithic Go API to a globally distributed, multi‑region service. How would you evolve the existing architecture to achieve high availability and horizontal scalability while minimizing disruption?
- 2You need to convince several teams to adopt a new service‑mesh and observability stack for Go services. What technical and organizational arguments would you make, and how would you handle migration of legacy services?
- 3Discuss the long‑term maintenance implications of choosing eventual consistency versus strong consistency for a globally replicated Go service, and how you’d design the system to allow future flexibility.
Follow-up Questions
- What metrics would you put in place to spot a failing instance quickly?
- How do you handle user session state when scaling horizontally?
- Can you compare the pros and cons of blue‑green versus canary deployments for this service?