Metrics in Observability

WHY IT EXISTS You cannot store and query every raw event forever at low cost, and for many questions you do not need to. Metrics exist to compress system behavior into compact numeric time series that are cheap to retain and instant to query, so you can watch trends and trigger alerts at scale.

THE MENTAL MODEL Think of a metric as a number that changes over time, tagged with a few low-cardinality labels. A counter only goes up, like total requests served. A gauge goes up and down, like current memory used or queue length. A histogram buckets observations, like request latency, so you can compute percentiles. The cost of this compactness is that the individual events are gone; you keep the shape, not the records.

HOW IT WORKS Instrumentation in the application increments counters, sets gauges, and records observations into histograms. An agent or scrape periodically collects these and stores them as time series in a database such as Prometheus. Queries aggregate across series and time windows, and alert rules evaluate expressions like the 99th percentile latency over five minutes. Labels let you split a metric by a small set of dimensions, but high-cardinality labels like user id explode the series count and are discouraged.

WHEN IT MATTERS Metrics are the right tool for the always-on view: dashboards, capacity trends, and fast alerting on golden signals like latency, traffic, errors, and saturation. They are the cheapest pillar, so they carry the broad real-time picture. When you then need to know exactly which request failed and why, you pivot to traces and logs, which retain per-event detail metrics deliberately discard.

ONE CONCRETE EXAMPLE A latency histogram shows the 99th percentile climbing from 200 to 900 milliseconds, and an error counter ticks up at the same time. That is enough to fire an alert and confirm a problem, but to find the exact slow requests you follow the alert into traces, because the histogram aggregated away the identity of each individual call.