What statistical methods automate canary-baseline comparison and handle noise?

Tests statistical rigor in automated canary-baseline comparison and robustness to production noise. A great answer covers non-parametric tests like Mann-Whitney U, aggregates many metrics with effect-size thresholds, rejects outliers via MAD, and uses smoothing or trimmed means to ignore spikes. It notes fresh baselines control startup effects and multiple comparison correction across metrics. Red flag: simple mean comparisons or raw p-value thresholds without effect sizes, outlier handling, or baseline isolation.