Guardrails for GitOps sync outages
layered safeguards around GitOps.
pre-merge schema validation, dry-run, policy gates and review; post-merge progressive sync, health checks with automated rollback, and pruning controls.
What's really being asked
Whether you think in layers: catch bad config before it merges, and contain damage after it does. A single gate is not enough.
The full answer
Pre-merge, in CI on the pull request: validate every manifest against the Kubernetes OpenAPI and CRD schemas, run a server-side dry-run apply against a real cluster to catch admission and conflict errors, lint with kubeval or similar, and enforce policy as code with OPA Gatekeeper or Kyverno tests so disallowed settings fail the build. Add required reviews and protected branches so nothing reaches main unreviewed. Render and diff Helm or Kustomize output so reviewers see the actual change. Post-merge, in the GitOps tool such as Argo CD or Flux: use progressive or staged rollout, attach health checks so a degraded resource pauses or self-heals the sync, configure automated rollback to the last healthy revision on a failed sync, and set prune and sync-window policies carefully so a bad delete is contained. Promote through environments so staging fails before production.
The mistakes people make
Relying solely on human code review; reviewers miss subtle misconfigurations. Adding only one check, like schema validation, and assuming it covers logic errors. Disabling auto-sync entirely, which trades safety for slow manual toil rather than adding guardrails. Forgetting the dry-run, which catches errors that static linting cannot.
What usually comes next
What is the difference between client-side and server-side dry-run? How does progressive delivery with health gates differ from a plain sync? How do you handle a bad change that passed all checks; what is your rollback and observability story? How do prune and self-heal settings cut both ways?
A concrete example
A pull request lowers a memory limit below the app's footprint. CI validates schema and policy fine, but the server-side dry-run plus a Kyverno rule requiring limits at least the documented minimum fails the build, blocking the merge. Had it slipped through, Argo CD's health check would see the pods OOMKill, halt the sync at the first environment, and roll back to the prior healthy revision before production was touched.
Interview question
Your team adds schema validation in CI after a GitOps outage. Why is that alone insufficient to prevent a similar incident?
- a.Schema validation should be replaced by disabling auto-sync entirely
- b.Schema validation cannot catch logic or resourcing errors; you also need dry-run, policy, and post-merge health gates with rollbackCorrect
- c.Schema validation already guarantees the cluster will be healthy after sync
- d.Post-merge checks are redundant once CI passes
Why? this is the answer
Schema validation only confirms structure, not whether a valid manifest behaves correctly. Layered guardrails, server-side dry-run, policy as code, and post-merge health checks with rollback, are needed because no single gate catches every failure.
Just read this? Test yourself on what you have been reading.
Read the original → github.com
- #gitops
- #argocd
- #ci-cd
- #policy-as-code
- #kubernetes
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on gitops — each one lists the topics its interview covers.
See open roles