How would you automate canary deployment and what metrics decide rollback?
This tests progressive delivery maturity. Strong answers cover traffic splitting (10% to 100%), automated 5-10 minute health gates, and rollback triggers like error rate and p99 latency.
What's really being asked
The interviewer wants to know if you understand progressive delivery as an operational practice, not just a feature flag. They are looking for your ability to automate safety gates, compare canary metrics against a stable baseline, and make rollback decisions data-driven rather than gut-driven. This separates engineers who deploy continuously from those who merely deploy frequently.
The full answer
First, traffic splitting mechanics. Start with a small percentage, such as 10 percent to the canary and 90 percent to stable, using a load balancer, service mesh, or Kubernetes-native selectors with replica ratios like 1 canary pod to 9 stable pods. Second, automated health gates. Each stage should hold for an analysis window, typically 5 to 10 minutes, to let metrics stabilize. Third, the decision engine. Use a metrics collector to compare canary against stable. If the canary passes, promote to the next stage, for example 25 percent, then 50 percent, then 100 percent. If it fails, automatically shift traffic back to stable and terminate the canary pods. Fourth, key metrics. Prioritize error rate spikes, p99 latency regression, throughput deviation, and saturation signals like CPU or memory. Include readiness and liveness probes as immediate pod-level gates before traffic even reaches the canary.
The mistakes people make
Proposing a manual go or no-go decision after eyeballing a dashboard. That does not scale and introduces human latency. Another red flag is monitoring only application logs without structured metrics or without comparing against the stable baseline. A third mistake is skipping pod health checks and relying solely on business metrics, which delays detection by minutes.
What usually comes next
How do you handle database schema changes during a canary? What happens if your metrics pipeline itself is delayed or drops data? How do you prevent a canary from exhausting shared resources like connection pools? How would you canary a stateful service?
A concrete example
In a Kubernetes environment, you run two Deployments, myapp-stable with nine replicas and myapp-canary with one replica. A load balancer sends 10 percent traffic to the canary. The canary pods must pass readiness probes on port 8080 before receiving traffic. You collect error rate and p99 latency for five minutes. If error rate stays within 0.1 percent of stable and p99 latency does not increase by more than 10 percent, the system automatically scales the canary to 3 replicas and shifts 25 percent traffic. If either threshold breaches, traffic reverts to stable and the canary deployment scales to zero.
Interview question
Why should automated canary pipelines run readiness and liveness probes before routing traffic to canary pods?
- a.They eliminate the need for automated rollback by guaranteeing canary pods are healthy before scaling
- b.They serve as the baseline metrics for comparing canary against stable performance
- c.They allow the analysis window to shrink from five minutes to under one minute without losing safety
- d.They catch pod-level failures immediately, preventing invalid traffic from delaying detection during the metrics analysis windowCorrect
Why? this is the answer
Readiness and liveness probes act as immediate pod-level gates that prevent traffic from reaching unhealthy canary instances, avoiding polluted metrics that would delay automated rollback by minutes. Option A is tempting because healthy pods seem sufficient, but probes cannot catch systemic regressions like latency spikes or elevated error rates that require metrics-based rollback.
Just read this? Test yourself on what you have been reading.
Read the original → oneuptime.com
- #ci/cd
- #canary deployment
- #kubernetes
- #sre
- #progressive delivery
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ci/cd — each one lists the topics its interview covers.
See open roles