Metrics in Observability
Metrics are numeric measurements aggregated over time, like counters, gauges, and histograms. They are cheap to store and fast to query, making them ideal for dashboards and alerting, but their pre-aggregation discards per-event detail needed for deep…
Why it exists
You cannot store and query every raw event forever at low cost, and for many questions you do not need to. Metrics exist to compress system behavior into compact numeric time series that are cheap to retain and instant to query, so you can watch trends and trigger alerts at scale.
The mental model
Think of a metric as a number that changes over time, tagged with a few low-cardinality labels. A counter only goes up, like total requests served. A gauge goes up and down, like current memory used or queue length. A histogram buckets observations, like request latency, so you can compute percentiles. The cost of this compactness is that the individual events are gone; you keep the shape, not the records.
How it works
Instrumentation in the application increments counters, sets gauges, and records observations into histograms. An agent or scrape periodically collects these and stores them as time series in a database such as Prometheus. Queries aggregate across series and time windows, and alert rules evaluate expressions like the 99th percentile latency over five minutes. Labels let you split a metric by a small set of dimensions, but high-cardinality labels like user id explode the series count and are discouraged.
When it matters
Metrics are the right tool for the always-on view: dashboards, capacity trends, and fast alerting on golden signals like latency, traffic, errors, and saturation. They are the cheapest pillar, so they carry the broad real-time picture. When you then need to know exactly which request failed and why, you pivot to traces and logs, which retain per-event detail metrics deliberately discard.
A concrete example
A latency histogram shows the 99th percentile climbing from 200 to 900 milliseconds, and an error counter ticks up at the same time. That is enough to fire an alert and confirm a problem, but to find the exact slow requests you follow the alert into traces, because the histogram aggregated away the identity of each individual call.
Interview question
What is the fundamental trade-off you accept by relying on metrics for observability?
- a.They are cheap and fast to query but discard per-event and high-cardinality detailCorrect
- b.They are only useful for logs, not for numeric values
- c.They retain full request traces but cannot be alerted on
- d.They are expensive to store but preserve every individual event
Why? this is the answer
Metrics aggregate data into compact time series, making them cheap and fast but stripping away individual events and high-cardinality detail. To recover that detail you use traces or logs, not metrics.
Just read this? Test yourself on what you have been reading.
- #metrics
- #observability
- #prometheus
- #sre
- #telemetry
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on metrics — each one lists the topics its interview covers.
See open roles