Diagnose a Prometheus cardinality explosion
operating Prometheus at scale.
find offenders via TSDB stats and topk count by __name__, identify unbounded labels, then drop or aggregate them with relabeling.
just scaling memory without fixing label design.
What's really being asked
Whether you understand Prometheus's data model, that every unique combination of metric name and label values is one time series, and that operational cost scales with the number of active series, not the number of scrapes.
The full answer
Start by confirming the symptom: check the TSDB status page or prometheus_tsdb_head_series for total active series, and prometheus_tsdb_head_samples_appended_total for ingestion rate. Run topk(10, count by (name)({name=~".+"})) to find which metrics produce the most series, then break down by label to find the offending dimension. The classic cause is a high-cardinality label: user ID, email, full URL path, pod name, container ID, or an error string attached as a label. In Kubernetes, ephemeral pod names rotating on every deploy compound this. Mitigation: use metric_relabel_configs to drop or replace offending labels at scrape time, aggregate paths into templated routes, move high-variance data into logs or traces instead of labels, and precompute with recording rules. Set sample and label limits per scrape to fail loudly.
The mistakes people make
Merely vertically scaling Prometheus memory, blaming query volume rather than series count, or suggesting more frequent scraping. Adding labels to fix it makes it worse.
What usually comes next
How do recording rules help? When would you shard or move to Thanos, Mimir, or Cortex? How do you enforce limits proactively?
A concrete example
A team added a user_id label to http_requests_total. With one million users, that single counter became one million time series. Active series jumped, memory ballooned, and PromQL slowed. A metric_relabel_config dropping user_id collapsed it back to a handful of series, and per-user analysis moved to traces.
Interview question
In Prometheus, what most directly drives the memory and query cost that a cardinality explosion inflates?
- a.The number of distinct active time series, one per unique label-setCorrect
- b.The number of Grafana dashboards querying the data
- c.The scrape interval configured for each target
- d.The retention window length in days
Why? this is the answer
Each unique metric-name-plus-label combination is a separate series held in the head block, so unbounded labels multiply series and cost. Scrape interval and dashboards affect load but are not what an unbounded-label explosion attacks.
Just read this? Test yourself on what you have been reading.
Read the original → grafana.com
- #prometheus
- #observability
- #cardinality
- #kubernetes
- #metrics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on prometheus — each one lists the topics its interview covers.
See open roles