Error Budgets: Balancing Reliability and Innovation

WHY IT EXISTS: Product teams want to ship features quickly, but operations teams need to maintain stability. This creates a natural tension, especially since new code changes are a primary source of outages, representing roughly 70% of incidents. Error budgets were created to resolve this conflict by providing a data-driven framework for balancing the pace of innovation with service reliability.

THE MENTAL MODEL: An error budget is the amount of unreliability you are willing to tolerate. It is not a goal to be spent, but a ceiling. The budget is calculated directly from your Service Level Objective (SLO): Error Budget = 1 - SLO. If your service has a 99.9% availability SLO, it has a 0.1% error budget. This means for every 1,000 requests, 1 can fail without violating the SLO. It transforms the abstract goal of "reliability" into a concrete, quantifiable budget that both developers and operations can use to make decisions.

HOW IT WORKS: An error budget policy defines the rules of engagement. While the budget has not been spent for a given period (e.g., four weeks), feature releases and other changes can proceed normally. Once the budget is exhausted by errors, the policy triggers an automatic response. Typically, this means a freeze on all non-essential releases, including new features. The team's focus is then redirected to reliability work—fixing bugs, improving monitoring, or addressing sources of instability—until the service is operating back within its SLO.

WHEN TO USE IT: Error budgets are essential for any service that has defined SLOs and a continuous release cycle. They provide an objective, non-confrontational way for teams to decide when to prioritize stability over new features. This is particularly valuable for services where development and operations are handled by different teams or where there's pressure to constantly ship.

WHEN THE POLICY'S 'FREEZE' DOESN'T APPLY: A release freeze is not a punishment and shouldn't be applied dogmatically. The policy should allow for exceptions. For example, if the error budget was consumed by a company-wide network outage, another team's faulty dependency (which is now also frozen), or by traffic that is out-of-scope for the SLO like load tests, the team may be permitted to continue feature work. The key is to use judgment and investigate the cause.

ONE CANONICAL EXAMPLE: A mobile game backend has a 99.9% availability SLO over a rolling 28-day window. In that period, it serves 10,000,000 requests. Its error budget is 0.1% of that total, which is 10,000 errors. If bugs, slow responses, or other failures cause more than 10,000 requests to fail within 28 days, the budget is spent. The pre-agreed policy then kicks in, halting all new game feature rollouts until the root causes are fixed and reliability returns to its target level.