Catching rare intermittent failures

WHAT THIS TESTS Whether you understand that intermittent, hard-to-reproduce failures are exactly what aggregate metric alerts miss, and that the fix is richer, event-level observability plus smarter alerting, not just lower thresholds.

A GOOD ANSWER COVERS The root issue is that coarse, averaged metrics smooth out rare failures: a handful of bad requests per thousand barely moves a fleet-wide average. So improve observability at the event level. Add distributed tracing so you can inspect the full path of individual failing requests, and use tail-based sampling so every error or unusually slow trace is retained rather than sampled away. Emit structured logs with rich, high-cardinality context, such as customer tier, region, version, and feature flags, so you can slice failures by dimension and spot a pattern, for instance failures only on one host, one shard, or one client version. Add exemplars to metrics so you can jump from a small bump in error rate straight to a captured failing trace. On the alerting side, move from coarse averages toward SLO error-budget burn-rate alerts, which catch slow steady degradation, and consider per-segment or anomaly-based detection so a rare but concentrated failure mode is visible. Together these let you catch and diagnose the needle in the haystack.

COMMON WRONG ANSWERS Just lowering existing thresholds, which adds noise without exposing the rare events. Relying on averages that mask outliers. Head-based sampling that discards the very traces you need. No high-cardinality context to slice by.

LIKELY FOLLOW-UPS Why tail sampling over head sampling here? How do exemplars help? How do you avoid cardinality blowups while keeping rich context in traces and logs?

ONE CONCRETE EXAMPLE A service fails roughly 1 in 500 requests, invisible in the average. You enable tail-based sampling to keep all error traces and add a version label to logs. Slicing reveals every failure comes from one canary build hitting a stale connection pool. An SLO burn-rate alert now fires on the concentrated error rate, and the captured traces pinpoint the bad build for rollback.