Design a 'Feature Flag' system for a high-traffic dashboard. How do you handle 'Configuration Drift'?

Question

Accepted Answer

A feature flag system for a high-traffic dashboard must balance low-latency evaluation, high availability, and real-time configuration updates. The architecture separates the control plane (where flags are created and managed) from the data plane (where flags are evaluated at runtime). Flags are cached aggressively at the edge to ensure that flag checks add negligible latency to page loads. Configuration drift—the divergence between intended flag configurations across environments or instances—is a critical risk that requires automated detection and reconciliation. The system consists of three core layers: a management UI and API for creating and updating flags; a configuration storage layer with versioned, audited flag definitions; and a runtime evaluation layer that serves flag values to application instances with sub-millisecond latency. Flag configurations are pushed from the control plane to a distributed cache (Redis Cluster) and further cached at the CDN edge or in application memory to minimize evaluation overhead. Feature Flag (Schema)
{
  "key": "dashboard_analytics_v2",
  "description": "Enable new analytics chart layout",
  "enabled": true,
  "version": 42,
  "created_at": "2025-01-15T10:00:00Z",
  "updated_at": "2025-03-20T14:30:00Z",
  "owner": "product-team@example.com",
  "conditions": {
    "percentage": 10,
    "user_segments": ["beta_users", "internal"],
    "environments": ["staging", "production"],
    "start_date": "2025-03-01T00:00:00Z"
  },
  "rollout_status": "gradual",
  "targeting_rules": [
    {
      "condition": "user.email contains @company.com",
      "value": true
    },
    {
      "condition": "user.id % 100 < 10",
      "value": true
    }
  ]
} 1. Application initializes SDK with configuration service endpoint and local cache settings 2. SDK maintains a background sync to fetch latest flag definitions every N seconds (configurable, e.g., 30s) 3. For each flag check, SDK evaluates in-memory cache (microsecond latency) 4. On cache miss or refresh cycle, SDK queries CDN-edge or local Redis cluster 5. Flag evaluations are logged asynchronously for analytics and debugging 6. All flag definitions include default values to handle service outages gracefully Configuration drift occurs when the intended flag configuration (defined in source control or the management UI) diverges from the actual configuration running in production. This can happen through manual changes across environments, failed deployments, or inconsistent application of updates. Drift risks include inconsistent user experiences, erroneous feature rollouts, and difficulty in debugging production issues. A mature feature flag system must provide visibility into drift and mechanisms to automatically correct it. **Version Control as Source of Truth**: Store flag configurations as code (YAML/JSON) in a Git repository. All changes require pull requests and approvals. The control plane continuously syncs with this repository, alerting on manual overrides. **Configuration Versioning**: Each flag carries a version number. The system tracks which version is deployed to each environment, allowing drift detection between environments. **Audit Trail**: Record every flag change (who, when, what) and every application sync (which instances received which version). This creates a complete history for post-incident analysis. **Automated Reconciliation**: Scheduled jobs compare intended configuration (from Git) with actual running configuration (from control plane). Differences trigger alerts and optionally auto-remediation. **Environment Tagging**: Each environment (dev, staging, prod) has distinct flag defaults. This prevents accidental production rollouts from staging configurations. **Canary Deployments**: Flag changes propagate first to a small subset of instances, then gradually roll out while monitoring error rates. Drift detectors compare canary versus control group flag values. **Health Endpoint**: Each application instance exposes its loaded flag versions via `/flags` endpoint, enabling external monitoring to verify consistent configuration across the fleet. # Reconciliation Job (Python pseudo-code)
def reconcile_flags():
    # Get desired state from Git
    desired_flags = load_flags_from_repo()
    
    # Get actual state from control plane API
    actual_flags = fetch_control_plane_flags()
    
    drift = {}
    for flag_key, desired in desired_flags.items():
        actual = actual_flags.get(flag_key)
        if actual and actual.version != desired.version:
            drift[flag_key] = {
                "desired_version": desired.version,
                "actual_version": actual.version,
                "diff": compare(desired, actual)
            }
    
    if drift:
        alert_pagerduty("Flag drift detected", drift)
        if AUTO_RECONCILE:
            for key, config in desired_flags.items():
                apply_flag(key, config)
        else:
            log_for_manual_review(drift)
    
    return dri

Design a 'Feature Flag' system for a high-traffic dashboard. How do you handle 'Configuration Drift'?

Scenario Questions

Follow-up Questions