Design automated canary analysis scoring

WHAT THIS TESTS: Your ability to design objective, automated canary scoring that safely gates releases without a human eyeballing dashboards.

KEY SLIS: Track the golden signals. Error rate, separating user-facing failures from benign ones. Latency at percentiles, especially p99, never just the average, because tails hide pain. Saturation such as CPU, memory, and connection-pool usage. Throughput normalized to traffic so a low-traffic canary is comparable. Add critical business or correctness metrics where they exist, like checkout success.

COMBINING INTO A SCORE: Compare the canary against a concurrent baseline running the old version under the same conditions, not against absolute targets, so environmental noise cancels out. For each metric, run a statistical comparison to judge whether the canary is meaningfully worse, mitigating small-sample noise. Assign weights reflecting importance, normalize each into a comparable scale, and aggregate into one health score. Define thresholds: above a high bar, auto-promote; below a low bar, auto-rollback; in the ambiguous middle, extend the bake time or escalate to a human.

COMMON WRONG ANSWERS: Scoring on raw averages, comparing the canary to fixed targets instead of a live baseline, ignoring statistical significance, or weighting all metrics equally regardless of impact.

LIKELY FOLLOW-UPS: How do you pick weights? How long should the canary bake? How do you avoid false rollbacks from noise? How do you handle metrics with different scales?

ONE CONCRETE EXAMPLE: The ACA system runs the canary and a baseline side by side for thirty minutes. It collects p99 latency, error rate, and CPU for both, runs a non-parametric test per metric, and finds error rate is significantly worse while latency and CPU are equivalent. Error rate carries the highest weight, so the combined score falls below the rollback threshold and the system automatically reverts and notifies the team, all without a human watching the dashboard.