On-Call Rotations: Engineering Reliability Under Pressure

WHY IT EXISTS To maintain service reliability and availability, systems need a rapid, human response to disruptions. Without a formal on-call process, services would suffer longer outages, failing to meet their availability goals. It formalizes the responsibility of responding to production incidents in a timely manner.

THE MENTAL MODEL Think of an on-call engineer not as a perpetual firefighter, but as a temporary guardian of production. Their role is time-bound and rotates through a team to share the load and prevent burnout. The ultimate goal for an on-call engineer is to use their operational insights to drive engineering projects that make their own future shifts quieter.

HOW IT WORKS An engineer is designated "on-call" for a specific period, carrying a device like a phone to receive alerts, or "pages." When an automated monitor detects an issue, it pages the on-call engineer. That engineer must acknowledge and respond within a pre-agreed time, such as 5 minutes for critical user-facing services or 30 minutes for less time-sensitive systems. Their job is to manage the incident, which may involve troubleshooting, rolling back a change, or escalating to others. The responsibility then rotates to the next person on the team.

WHEN TO USE IT On-call rotations are essential for any service where high availability is a business requirement. This is especially true for user-facing systems like e-commerce sites or critical backend infrastructure where downtime has a significant financial or reputational impact. The required response time is directly tied to the service's availability target (SLO); a 99.99% available service can only tolerate about 13 minutes of downtime per quarter, demanding a response in minutes.

WHEN NOT TO USE IT On-call is a significant operational cost and a source of stress. It is not appropriate for services without clear reliability targets or where occasional downtime is acceptable. It is also a recipe for burnout in teams that are purely operational and lack the mandate or skills to engineer away the sources of pages. If a team only ever fights fires, they will never have time to build fire-resistant systems.

ONE CANONICAL EXAMPLE Google's SRE teams for services like Search and Gmail use on-call rotations. An SRE on-call for a user-facing system is expected to respond to a page within 5 minutes. To ensure this doesn't become an overwhelming burden, Google caps the time an SRE can spend on operational work (like handling pages) at 50%. The other 50% of their time must be spent on engineering projects that improve service reliability and automate tasks, with the explicit goal of reducing future pages.