What is configuration drift in GitOps and how do tools handle it?

WHAT THIS TESTS: The interviewer wants to know if you grasp the fundamental GitOps contract: the Git repository is the single source of truth, the live cluster is the observed state, and the tool in the middle is a reconciler. They are checking whether you see drift as a failure of that contract, not just a minor ops annoyance. At the senior level, they also want to hear a spectrum of enforcement strategies and the tradeoffs between them.

A GOOD ANSWER COVERS: First, define configuration drift precisely as any change to the live system that is not reflected in the Git history. This includes manual kubectl edits, automated scripts, or compromised containers mutating their own environments. Second, explain why it is a problem: it breaks auditability because the Git history no longer describes reality, it destroys repeatability because new clusters cannot be rebuilt from the repo, and it creates toil because operators must reverse-engineer live state before making planned changes. Third, lay out the strategies a GitOps tool can use. The softest is read-only drift detection that surfaces alerts without acting. The middle path is automated self-healing sync where the controller continuously reverts deviations back to the Git definition. The hardest is prevention through admission control or RBAC that blocks out-of-band writes at the Kubernetes API level. Fourth, mention that mature teams often combine these layers: detection for visibility, sync for resilience, and admission for guardrails.

COMMON WRONG ANSWERS: Suggesting that operators should fix drift by running manual kubectl commands to align the cluster with Git inverts the workflow and creates more drift. Claiming that drift is only about accidental changes misses the security angle, since attackers and rogue automation also cause it. Saying the solution is to disable the reconciler when it fights with the team signals a fundamental misunderstanding of the GitOps model. Confusing drift with normal rolling updates or HPA scaling is another basic error, because those are controller-driven changes that should be defined in Git rather than classified as drift.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a break-glass scenario where someone must mutate the cluster during an incident. They may ask about the performance cost of continuous reconciliation at scale. They may also ask how to handle secrets or external state stores that cannot live in plain Git, since those are common sources of pseudo-drift. Another angle is comparing pull-based versus push-based GitOps and which model handles drift more naturally.

ONE CONCRETE EXAMPLE: Imagine a platform team uses Argo CD to manage a namespace. An on-call engineer runs kubectl set image during an outage to roll forward a buggy deployment. The change is never committed to Git. Two days later, another engineer applies a routine manifest update from Git, but Argo CD has been configured with automated sync and prune. The tool detects the out-of-band image change, reverts it to the Git-defined version, and the buggy image returns to production because the hotfix was never recorded in the repository. This illustrates why drift is dangerous and why teams need either a policy that blocks manual edits or a strict post-incident process to backport emergency changes into Git within minutes.