tezvyn:

SLO Burn Rate: Alerting on Budget Consumption

AI-drafted, machine-checkedSource: sre.googleintermediate

SLO burn rate measures how fast you're using your error budget. A burn rate of 2 means you're consuming budget twice as fast as allowed. It's used to trigger alerts for both fast outages and slow-burning problems to prevent SLO breaches.

WHY IT EXISTS: Service Level Objectives (SLOs) are useless without a strategy to defend them. Simply alerting when an SLO is about to be breached is too late. SLO burn rate alerting was developed to provide early, actionable warnings about significant events that threaten the error budget, balancing speed of detection with alert precision.

THE MENTAL MODEL: Think of your error budget for a month as a full tank of gas for a long road trip. The SLO burn rate is your car's real-time fuel consumption. A burn rate of 1 means you're driving efficiently and will finish the trip with an empty tank. A burn rate of 10 means you're flooring it and will run out of gas long before your destination, leaving you stranded.

HOW IT WORKS: Burn rate is the multiple of how fast you're consuming your error budget compared to the sustainable rate. It's calculated by dividing the current error rate (over a short window) by the target error rate defined by your SLO. For a 99.9% availability SLO over 30 days, the sustainable error rate is 0.1%. If your service sees a 2% error rate over the last 5 minutes, the burn rate is 2% / 0.1% = 20. This means you are consuming your budget 20 times faster than planned.

WHEN TO USE IT: Use burn rate alerting to create high-fidelity alerts for any service with an SLO. It's especially effective for creating a multi-tiered alerting strategy. For example, you can create a critical page for a very high burn rate over a short window (e.g., "we'll burn 5% of our monthly budget in the next 2 hours") and a non-urgent ticket for a low burn rate over a long window (e.g., "we'll burn our entire budget in 2 weeks if this continues").

WHEN NOT TO USE IT: Burn rate alerting can be overly sensitive for services with very low or highly sporadic traffic. A handful of errors during a low-traffic minute can trigger a massive burn rate spike, leading to false alarms. These systems may require adding a minimum request threshold to the alert condition to improve precision.

ONE CANONICAL EXAMPLE: A naive approach is to alert if the error rate in the last 10 minutes exceeds the overall SLO threshold (e.g., 0.1% for a 99.9% SLO). This is a burn rate of 1. This is a poor strategy because it's too slow for a total outage and too noisy for minor blips. A much better strategy is to alert on a high burn rate, like 14.4 over a 60-minute window. This corresponds to consuming 2% of a 30-day error budget in one hour, indicating a significant event that requires immediate attention.

Read the original → sre.google

Get five bites like this every day.

Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.