The Four Golden Signals of Service Monitoring

WHY IT EXISTS: Modern systems expose thousands of potential metrics, creating a needle-in-a-haystack problem during an outage. The Four Golden Signals were developed as a simple, opinionated framework to cut through the noise and focus on what actually matters for service health from a user's perspective.

THE MENTAL MODEL: Think of the Four Golden Signals as the vital signs for a software service, much like a doctor checks heart rate, blood pressure, and respiration. They provide a quick, high-level assessment of your service's health. If these signals are good, your users are likely having a good experience.

HOW IT WORKS: The framework consists of four key metrics to monitor for any service. First, Latency: The time it takes to service a request. It's critical to distinguish the latency of successful requests from that of failed requests, as a fast error is still an error. Second, Traffic: A measure of demand on your system, such as requests per second for a web service. Third, Errors: The rate of requests that fail, including explicit failures like HTTP 5xx codes and implicit ones like a 200 OK response with wrong content. Fourth, Saturation: How "full" your service is, measuring your most constrained resource like CPU or memory. It's a leading indicator of future problems.

WHEN TO USE IT: These signals should be the primary source for your highest-level dashboards and alerts. They are the ideal metrics for defining Service Level Objectives (SLOs) because they directly represent the user experience. Use them to answer the question, "Is the service healthy right now?"

WHEN NOT TO USE IT: The Golden Signals are not a replacement for detailed, low-level system metrics. They tell you that a problem exists (e.g., high latency) but not necessarily why (e.g., a specific database query is slow). You still need deeper, white-box metrics for debugging and root cause analysis.

ONE CANONICAL EXAMPLE: An on-call engineer gets an alert for high latency. They check the Golden Signals dashboard. Latency is spiking, but Traffic is flat and Errors are low. They then look at Saturation and see that memory utilization is at 98%, causing frequent garbage collection pauses. Saturation was the leading indicator that explained the latency symptom.