Guardrails for GitOps sync outages

WHAT THIS TESTS Whether you think in layers: catch bad config before it merges, and contain damage after it does. A single gate is not enough.

A GOOD ANSWER COVERS Pre-merge, in CI on the pull request: validate every manifest against the Kubernetes OpenAPI and CRD schemas, run a server-side dry-run apply against a real cluster to catch admission and conflict errors, lint with kubeval or similar, and enforce policy as code with OPA Gatekeeper or Kyverno tests so disallowed settings fail the build. Add required reviews and protected branches so nothing reaches main unreviewed. Render and diff Helm or Kustomize output so reviewers see the actual change. Post-merge, in the GitOps tool such as Argo CD or Flux: use progressive or staged rollout, attach health checks so a degraded resource pauses or self-heals the sync, configure automated rollback to the last healthy revision on a failed sync, and set prune and sync-window policies carefully so a bad delete is contained. Promote through environments so staging fails before production.

COMMON WRONG ANSWERS Relying solely on human code review; reviewers miss subtle misconfigurations. Adding only one check, like schema validation, and assuming it covers logic errors. Disabling auto-sync entirely, which trades safety for slow manual toil rather than adding guardrails. Forgetting the dry-run, which catches errors that static linting cannot.

LIKELY FOLLOW-UPS What is the difference between client-side and server-side dry-run? How does progressive delivery with health gates differ from a plain sync? How do you handle a bad change that passed all checks; what is your rollback and observability story? How do prune and self-heal settings cut both ways?

ONE CONCRETE EXAMPLE A pull request lowers a memory limit below the app's footprint. CI validates schema and policy fine, but the server-side dry-run plus a Kyverno rule requiring limits at least the documented minimum fails the build, blocking the merge. Had it slipped through, Argo CD's health check would see the pods OOMKill, halt the sync at the first environment, and roll back to the prior healthy revision before production was touched.