How would you design observability — metrics, logs, and traces — for a Go microservice in production?

Question

Accepted Answer

// Metrics with Prometheus
import "github.com/prometheus/client_golang/prometheus"

var (
    requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "path", "status"})

requestsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
    }, []string{"method", "path", "status"})
)

func metricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        rw := &responseRecorder{ResponseWriter: w, code: 200}
        next.ServeHTTP(rw, r)
        duration := time.Since(start).Seconds()
        status := strconv.Itoa(rw.code)
        requestDuration.WithLabelValues(r.Method, r.URL.Path, status).Observe(duration)
        requestsTotal.WithLabelValues(r.Method, r.URL.Path, status).Inc()
    })
}

// OpenTelemetry tracing
import "go.opentelemetry.io/otel"

func (s *UserService) GetUser(ctx context.Context, id string) (*User, error) {
    ctx, span := otel.Tracer("user-service").Start(ctx, "GetUser")
    defer span.End()

span.SetAttributes(attribute.String("user.id", id))

user, err := s.repo.FindByID(ctx, id)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return nil, err
    }
    return user, nil
} Metrics: request rate, error rate, latency (P50/P95/P99), goroutine count, GC pause time Logs: structured JSON with traceID, spanID, requestID, userID on every entry Traces: OpenTelemetry spans for HTTP requests, DB queries, outbound calls — visualized in Jaeger/Tempo Correlate by injecting traceID into log context: span.SpanContext().TraceID().String() Stack: Prometheus + Grafana (metrics), Loki (logs), Tempo/Jaeger (traces) — or Datadog/New Relic for managed

How would you design observability — metrics, logs, and traces — for a Go microservice in production?

Expose Prometheus metrics on /metrics, emit structured JSON logs with slog enriched with trace IDs, and instrument with OpenTelemetry for distributed traces. Correlate all three with a shared trace ID.

How would you design observability — metrics, logs, and traces — for a Go microservice in production?

Expose Prometheus metrics on /metrics, emit structured JSON logs with slog enriched with trace IDs, and instrument with OpenTelemetry for distributed traces. Correlate all three with a shared trace ID.

Scenario Questions

Follow-up Questions