Automate a canary release with a 1% 5xx error threshold

WHAT THIS TESTS: This question evaluates whether you understand progressive delivery as a control loop rather than a deployment script. The interviewer wants to see that you can translate a business-level SLO, less than 1% 5xx errors, into concrete integrations between a rollout orchestrator, a traffic plane, and an observability backend. Senior candidates should demonstrate awareness of automation safety, feedback latency, and separation of concerns.

A GOOD ANSWER COVERS: First, traffic shifting mechanics. The tool does not route traffic itself; it manipulates a service mesh like Istio or an ingress controller like NGINX to move a small percentage of live traffic to the new canary pods. Second, metric definition and querying. You define an AnalysisTemplate that specifies the query, for example a PromQL expression measuring 5xx rate over a two-minute window, and the success threshold, in this case a value below 0.01. Third, the decision loop. The controller creates an AnalysisRun that executes the query at intervals; if the metric stays within bounds for the required duration and sample count, the controller automatically promotes the canary to full traffic, but if the 5xx rate exceeds 1% or the failure limit is reached, it aborts and rolls back traffic to the stable version. Fourth, required integrations. Besides Kubernetes, you need a metrics provider such as Prometheus or Datadog, a traffic management layer capable of weighted splits, and optionally a notification system to alert teams when an analysis fails.

COMMON WRONG ANSWERS: A major red flag is stating that Argo Rollouts or Flagger directly collects application logs or metrics from the pods. These controllers are orchestrators; they read from external observability systems. Another weak pattern is describing a canary that simply deploys new pods and waits for a human to decide, which ignores the automated analysis requirement. Candidates also err by omitting the traffic management component entirely, suggesting that a canary works by simply scaling up the new replica set without splitting user traffic, which would expose all users to a potentially broken build.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle metric flapping or noisy signals, which leads to discussing consecutive success limits versus failure limits and longer analysis windows. They might probe what happens if Prometheus is unavailable during the rollout, which tests whether you consider dry-run modes or fallback abort policies. Another angle is blue-green versus canary trade-offs, or how to run pre-promotion analysis before any traffic hits the canary.

ONE CONCRETE EXAMPLE: Imagine a Rollout resource with a canary strategy set to send 5% of traffic to the new version via an Istio VirtualService. The AnalysisTemplate named error-rate queries Prometheus for sum(rate(http_requests_total{status=~"5.."}[2m])) / sum(rate(http_requests_total[2m])). The success condition requires the result to be less than 0.01. The controller starts the AnalysisRun, shifts traffic, and evaluates the query every minute. After three consecutive successful measurements, it promotes the Rollout to 100% and scales down the old version. If the 5xx rate hits 2% on the second measurement, the failure limit is reached, traffic reverts to stable, and the new pods are scaled to zero.