Advanced interview questions in DevOps & Cloud, page 7
Schema evolution without rewriting history
Use a table format with metadata-level evolution, add a new column rather than mutating the old, and reconcile types at read time; avoid rewriting petabytes.
Describe a robust strategy for GitOps database schema migrations
Tests imperative-to-declarative schema reconciliation. Strong answers version idempotent pre-sync jobs and colocate schema state in Git. They use dedicated operators, not infra tools, for live execution.
How would you use GitOps to manage Kubernetes cluster lifecycles?
This tests cluster lifecycle GitOps, not just app delivery. A strong answer uses Flux plus Cluster API on a hub cluster, stores cluster definitions in Git, and rolls upgrades via MachineDeployments.
How do you architect self-service GitOps with central policy enforcement?
What it tests: balancing self-service GitOps with hard guardrails. A strong answer covers namespace-scoped controllers, OPA/Kyverno deny policies, and a centralized policy repo. Red flag: proposing manual ticket reviews instead of automated admission gates.
Diagnose a Prometheus cardinality explosion
Find offenders via TSDB stats and topk count by __name__, identify unbounded labels, then drop or aggregate them with relabeling.
Head-based vs tail-based trace sampling
Head decides up front (cheap, may miss rare errors); tail decides after the trace completes (catches errors and slow traces but needs buffering).
Design auto drift detection and retraining
Capture inputs and predictions, compute data and concept drift metrics on a schedule, alert on threshold breach, and trigger a retraining and redeploy pipeline.
Chaos test for gray-failure cascades in shared services?
Inject partial latency into a shared service, hypothesize tenants stay isolated within steady state, and monitor cross-system queue depth, pool saturation, retries, and per-tenant SLIs.
Design petabyte-scale distributed training
Object storage with columnar formats, distributed preprocessing, a data-parallel framework with efficient sharded loading, and managed orchestration.
Client-side chaos for an uncontrollable third party?
Inject faults at your client boundary via a proxy or fault-injecting wrapper, simulate timeouts, errors, and latency, then verify timeouts, retries, breakers, and fallbacks.
Design a secure multi-tenant CI/CD runner on Kubernetes
Apply namespaces, NetworkPolicies, Pod Security Standards; cap resources with ResourceQuotas and LimitRanges; schedule to dedicated or sandboxed nodes.
How would you architect deployment capabilities for multiple disparate environments?
This tests platform design for secure multi-environment orchestration. A strong answer proposes a declarative control plane, environment-local agents with short-lived credentials, and templating for consistency.
How would you modify CI/CD to generate and sign SLSA provenance?
Tests SLSA Build Track L1-L3 architecture. Outline: emit in-toto provenance with builder ID and resolved deps from ephemeral runners; sign via DSSE with Sigstore; publish attestations to registry. Red flag: confusing SBOMs or image signatures with provenance.
Technical challenges of a multi-cloud strategy
Data consistency and egress costs across providers, cross-cloud networking and latency, and federating disparate IAM systems, plus operational and tooling overhead.
GitOps repo layout for environment promotion
Shared base plus per-env overlays via Kustomize or value files, promotion by PR moving a pinned version forward, separating app source from config repos.
Design an enterprise cloud landing zone
Multi-account or subscription structure, centralized identity and SSO, network topology like hub-and-spoke, guardrails via policy and SCPs, and centralized logging.
Helm migration hooks under GitOps
Use a pre-upgrade hook Job with weights and delete policy; the challenge is GitOps tools render statically and reconcile, conflicting with Helm's imperative hook lifecycle.
Designing an error budget policy
Define SLO and budget, tiered consequences as burn worsens, a feature freeze on exhaustion, and concrete earn-back criteria.
Migrate an OLTP database with minimal downtime
Take an initial bulk load, then use change data capture to replicate ongoing changes until source and target are in sync, validate, then cut over during a brief window with a rollback plan.
Keeping operator .status accurate under failures
Status can lag or go stale during partitions and crashes; make reconcile idempotent, observe true state each loop, use conditions and observedGeneration, handle conflicts.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles