How would you automate canary deployment and what metrics decide rollback?

WHAT THIS TESTS: The interviewer wants to know if you understand progressive delivery as an operational practice, not just a feature flag. They are looking for your ability to automate safety gates, compare canary metrics against a stable baseline, and make rollback decisions data-driven rather than gut-driven. This separates engineers who deploy continuously from those who merely deploy frequently.

A GOOD ANSWER COVERS: First, traffic splitting mechanics. Start with a small percentage, such as 10 percent to the canary and 90 percent to stable, using a load balancer, service mesh, or Kubernetes-native selectors with replica ratios like 1 canary pod to 9 stable pods. Second, automated health gates. Each stage should hold for an analysis window, typically 5 to 10 minutes, to let metrics stabilize. Third, the decision engine. Use a metrics collector to compare canary against stable. If the canary passes, promote to the next stage, for example 25 percent, then 50 percent, then 100 percent. If it fails, automatically shift traffic back to stable and terminate the canary pods. Fourth, key metrics. Prioritize error rate spikes, p99 latency regression, throughput deviation, and saturation signals like CPU or memory. Include readiness and liveness probes as immediate pod-level gates before traffic even reaches the canary.

COMMON WRONG ANSWERS: Proposing a manual go or no-go decision after eyeballing a dashboard. That does not scale and introduces human latency. Another red flag is monitoring only application logs without structured metrics or without comparing against the stable baseline. A third mistake is skipping pod health checks and relying solely on business metrics, which delays detection by minutes.

LIKELY FOLLOW-UPS: How do you handle database schema changes during a canary? What happens if your metrics pipeline itself is delayed or drops data? How do you prevent a canary from exhausting shared resources like connection pools? How would you canary a stateful service?

ONE CONCRETE EXAMPLE: In a Kubernetes environment, you run two Deployments, myapp-stable with nine replicas and myapp-canary with one replica. A load balancer sends 10 percent traffic to the canary. The canary pods must pass readiness probes on port 8080 before receiving traffic. You collect error rate and p99 latency for five minutes. If error rate stays within 0.1 percent of stable and p99 latency does not increase by more than 10 percent, the system automatically scales the canary to 3 replicas and shifts 25 percent traffic. If either threshold breaches, traffic reverts to stable and the canary deployment scales to zero.

Source: oneuptime.com