Cloud Monitoring: Metrics, Time Series, and Resources

WHY IT EXISTS: Cloud platforms are complex, distributed systems. To operate them reliably, you need visibility into their health and performance. Without quantitative measurements, you cannot know if a service is working correctly, debug why it's slow, or plan for future capacity needs.

THE MENTAL MODEL: Think of monitoring data as a three-part structure: a metric type, a monitored resource, and a time series. The metric type is what you are measuring (e.g., CPU utilization). The monitored resource is which thing you are observing (e.g., a specific virtual machine instance). The time series is the actual data: a sequence of timestamped values for that metric on that resource.

HOW IT WORKS: Data is collected from various sources. Agents, like the Ops Agent, run on VMs to gather system metrics and data from third-party applications like Apache or Redis. Services can also emit metrics directly, or you can instrument your own applications using APIs like OpenTelemetry to create user-defined metrics. Each data point is tagged with labels that identify the specific resource. This data is stored in a time-series database. You interact with it by filtering to select specific series (e.g., all VMs in one zone) and aggregating to combine data (e.g., calculating the average CPU utilization across those VMs).

WHEN TO USE IT: Use metrics as the foundation for your observability strategy. Three primary uses are: first, creating dashboards for real-time visualization of system health; second, building alerting policies that notify you of problems, like when a metric crosses a threshold or is absent for too long; third, analyzing historical data to debug incidents or perform capacity planning. This applies to infrastructure (VMs, databases, networks) and custom application performance.

WHEN NOT TO USE IT: Metrics are for aggregated, numerical data over time. They are not the right tool for capturing high-cardinality, arbitrary event data like individual user actions or full error stack traces; use structured logs for that. Metrics also don't replace distributed tracing, which is designed to follow a single request's path through multiple services. Metrics tell you the "what" (high latency), while logs and traces help you find the "why" (a specific slow database query).

ONE CANONICAL EXAMPLE: An alerting policy for a web server fleet. The metric type could be nginx/requests/total. You would filter for all server instances in your production environment. You could then aggregate these time series by summing them to get the total request rate. The alerting policy might be: "If the total request rate drops by 50% compared to the previous week's average for more than 5 minutes, send a notification to the on-call channel."