Configuration Drift: When Live State Betrays Git

WHY IT EXISTS: Manual changes to live environments, like using kubectl edit, break the "single source of truth" principle of GitOps. This "drift" makes systems unpredictable, complicates rollbacks, and erodes the audit trail. Drift detection was created to automatically find and flag these out-of-band modifications.

THE MENTAL MODEL: Think of drift detection as an automated audit. Your Git repository is the official blueprint for your application's infrastructure. A controller, like Argo CD, is the inspector who constantly compares the live running application against that blueprint. Any deviation, from a changed replica count to a modified config map, is immediately reported.

HOW IT WORKS: A GitOps controller continuously performs a comparison loop. First, it fetches the desired state defined in your Git repository's manifests (YAML, Helm charts, etc.). Second, it queries the Kubernetes API server for the current, live state of those same resources. Third, it performs a "diff" between the desired state and the live state. If any differences are found, the application is marked as having drifted (e.g., Argo CD shows an "OutOfSync" status).

WHEN TO USE IT: Drift detection is fundamental to any GitOps workflow. Use it to enforce infrastructure-as-code discipline, ensure your cluster's state is always auditable and reproducible from Git, and prevent manual hotfixes from becoming permanent, untracked liabilities. It's key for maintaining stability and compliance.

WHEN NOT TO USE IT: The concept is less useful in imperative workflows where there is no single declarative source of truth to compare against. More importantly, you must configure it to ignore certain fields that are expected to change. For example, Kubernetes itself will add a clusterIP to a Service manifest after it's created. Without a proper ignore rule, this will be flagged as drift, creating constant false positives.

ONE CANONICAL EXAMPLE: A developer's Git manifest for a deployment specifies replicas: 2. To handle a temporary load, an operator manually runs kubectl scale deployment my-app --replicas=4. Argo CD, on its next check, will see the live replica count (4) does not match the desired count in Git (2). It will flag the application as "OutOfSync". Depending on its configuration, it might simply report the drift or automatically "heal" the application by scaling it back down to 2 replicas.