Diagnose a degraded canary release

WHAT THIS TESTS: Your structured judgment under uncertainty, deciding whether a small canary regression is signal or noise and whether it justifies rollback.

A GOOD ANSWER COVERS: Begin by validating the data. Is a two percent p99 rise and half-percent error bump statistically significant given the canary's smaller sample, or within normal variance? Confirm the comparison is fair: the canary and baseline should see the same traffic mix and the canary should not be penalized by cold caches or a tiny instance pool. Next, localize the regression. Slice errors by endpoint, status code, and dependency, and use distributed traces to see whether latency grew in your code or in a downstream call. Inspect resource metrics for CPU, memory, or connection-pool saturation. Then decide against the SLO and remaining error budget: weigh user impact and trend direction, not just the raw delta.

COMMON WRONG ANSWERS: Rolling back instantly on any wiggle, proceeding while errors trend upward, or comparing the canary to a non-equivalent baseline and drawing false conclusions.

LIKELY FOLLOW-UPS: How do you tell significance from noise with small samples? Which errors are user-facing versus benign? How long do you bake the canary before deciding? What automated thresholds would you set?

ONE CONCRETE EXAMPLE: You discover via traces that the canary's extra latency and errors all come from a single endpoint making a new call to a downstream service whose connection pool is undersized in the canary config. The delta is real and trending worse as traffic ramps. Because it would burn error budget and affect real users, you roll back, fix the pool sizing, and recanary. Had the deltas instead been within the noise band of a tiny canary and flat over time, you would proceed, since reflexive rollback on noise just slows safe delivery.