Define configuration drift in IaC. How do you detect and remediate it?

WHAT THIS TESTS: This question probes your understanding of the operational reality that infrastructure state diverges from its declared configuration over time. The interviewer wants to see if you recognize that configuration drift is not merely an academic inconsistency but a primary cause of disaster recovery and high availability failures. They are looking for maturity in observability, closed-loop remediation, and governance that treats the IaC repository as the single source of truth rather than a one-time provisioning script.

A GOOD ANSWER COVERS: First, a precise definition drawn from the source material: configuration drift is the gradual divergence between an initially configured primary infrastructure and its supposedly identical secondary or peer configurations, driven by ongoing ad hoc changes. Second, proactive detection strategies such as scheduled state comparisons using tooling like Terraform plan or CloudFormation drift detection, continuous policy scanning with AWS Config or OPA, and runtime anomaly detection that catches manual console changes within minutes rather than months. Third, remediation strategies ranked by aggressiveness: automated reconciliation where the tool reverts unauthorized changes; immutable redeployment that replaces mutated resources rather than patching them; and breaking-glass workflows that capture justified manual changes and route them back into version control. Fourth, preventive guardrails including least-privilege cloud access, GitOps workflows that block direct production changes, and mandatory code review for all infrastructure modifications.

COMMON WRONG ANSWERS: A weak answer treats drift as a legacy-only problem or suggests quarterly manual audits as sufficient detection. Another red flag is recommending root cause analysis without an automated feedback loop, which leaves the environment exposed during investigation. Candidates who propose fixing drift by editing resources directly in the console and later updating code demonstrate a fundamental misunderstanding of source-of-truth discipline. Finally, ignoring the disaster recovery implication, specifically that drift between primary and secondary environments accounts for most DR and HA system failures, signals shallow operational experience.

LIKELY FOLLOW-UPS: The interviewer may ask how you balance auto-remediation against the risk of destroying legitimate emergency changes made during an outage. They might probe cost and scale concerns, such as how to scan for drift across thousands of accounts without excessive API call costs. Another common thread is Kubernetes-specific drift, where controllers, admission webhooks, and node autoscaling create constant state churn that naive detection tools flag as false positives. You should also be ready to discuss how to handle drift in brownfield environments where IaC was introduced after resources already existed.

ONE CONCRETE EXAMPLE: Imagine an on-call engineer opens an AWS security group rule directly in the console at 2 AM to restore connectivity during an incident. The Terraform state now diverges from reality. A proactive detection pipeline running Terraform plan every fifteen minutes flags the extra ingress rule within one cycle. The remediation strategy triggers an automated revert of the security group to the declared state, while simultaneously paging the team to confirm whether the change should be codified in the module or was truly an emergency exception. If left unremediated, this single rule difference replicates into disaster recovery failures because the secondary region, provisioned from the clean Terraform code, lacks the same network path, causing failover traffic to drop when the primary fails.

Source: wiki.gccollab.ca