Fixing a noisy non-actionable alert

WHAT THIS TESTS Whether you treat every page as something that must be actionable and approach alert noise as a data-driven tuning problem, not a nuisance to mute.

A GOOD ANSWER COVERS Start by quantifying the problem: pull the alert's firing history and, for each fire, determine whether a human actually had to do something. This actionability rate is the key metric. If the alert is almost never actionable, you have options. If it represents a real but non-urgent condition, downgrade it from a page to a ticket or a dashboard panel. If it is firing on transient blips, add or increase a for duration so it only fires when the condition persists, and revisit the threshold so it reflects genuine impact rather than normal variance. Best of all, re-anchor it to a user-facing symptom or SLO so it fires when users are affected, not when an internal metric wiggles. If it provides no value at all, delete it; a deleted noisy alert is a legitimate outcome. Finally, monitor the change to confirm noise dropped without missing real incidents, and treat this as a recurring hygiene practice.

COMMON WRONG ANSWERS Just silencing or snoozing it indefinitely, which hides real failures. Bumping the threshold arbitrarily without checking what is normal. Refusing to delete because someone might want it. Ignoring that the root issue may be that it is a cause alert, not a symptom alert.

LIKELY FOLLOW-UPS How do you define actionability? When is deleting an alert the right call? How does the for clause reduce flapping?

ONE CONCRETE EXAMPLE An alert fires whenever a single node's CPU exceeds 80 percent, several times daily, yet the service is always fine due to redundancy. History shows zero of the last 60 fires needed action. You delete the page, keep CPU on the dashboard, and add an SLO-based alert on actual request latency. Pages drop sharply and real degradations still surface.