Design automated canary analysis scoring
Automated release gating.
track the golden-signal SLIs, compare canary to baseline statistically, weight and combine into a score with promote/rollback thresholds.
What's really being asked
Your ability to design objective, automated canary scoring that safely gates releases without a human eyeballing dashboards.
Key slis
Track the golden signals. Error rate, separating user-facing failures from benign ones. Latency at percentiles, especially p99, never just the average, because tails hide pain. Saturation such as CPU, memory, and connection-pool usage. Throughput normalized to traffic so a low-traffic canary is comparable. Add critical business or correctness metrics where they exist, like checkout success.
Combining into a score
Compare the canary against a concurrent baseline running the old version under the same conditions, not against absolute targets, so environmental noise cancels out. For each metric, run a statistical comparison to judge whether the canary is meaningfully worse, mitigating small-sample noise. Assign weights reflecting importance, normalize each into a comparable scale, and aggregate into one health score. Define thresholds: above a high bar, auto-promote; below a low bar, auto-rollback; in the ambiguous middle, extend the bake time or escalate to a human.
The mistakes people make
Scoring on raw averages, comparing the canary to fixed targets instead of a live baseline, ignoring statistical significance, or weighting all metrics equally regardless of impact.
What usually comes next
How do you pick weights? How long should the canary bake? How do you avoid false rollbacks from noise? How do you handle metrics with different scales?
A concrete example
The ACA system runs the canary and a baseline side by side for thirty minutes. It collects p99 latency, error rate, and CPU for both, runs a non-parametric test per metric, and finds error rate is significantly worse while latency and CPU are equivalent. Error rate carries the highest weight, so the combined score falls below the rollback threshold and the system automatically reverts and notifies the team, all without a human watching the dashboard.
Interview question
In automated canary analysis, what is the correct reference point for judging whether the canary is healthy?
- a.The team's average latency across all services
- b.Fixed absolute SLO targets defined months ago
- c.A concurrent baseline running the old version under the same conditionsCorrect
- d.The canary's own metrics from the previous deployment
Why? this is the answer
Comparing the canary to a concurrent baseline cancels out environmental noise like traffic shifts and shared-dependency hiccups. Fixed targets do not adapt to current conditions, so they trigger false promotions or rollbacks when the baseline itself moves.
Just read this? Test yourself on what you have been reading.
Read the original → sre.google
- #canary-analysis
- #sli
- #release-automation
- #statistics
- #deployment
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles