Diagnose a Prometheus cardinality explosion

WHAT THIS TESTS Whether you understand Prometheus's data model, that every unique combination of metric name and label values is one time series, and that operational cost scales with the number of active series, not the number of scrapes.

A GOOD ANSWER COVERS Start by confirming the symptom: check the TSDB status page or prometheus_tsdb_head_series for total active series, and prometheus_tsdb_head_samples_appended_total for ingestion rate. Run topk(10, count by (__name__)({__name__=~".+"})) to find which metrics produce the most series, then break down by label to find the offending dimension. The classic cause is a high-cardinality label: user ID, email, full URL path, pod name, container ID, or an error string attached as a label. In Kubernetes, ephemeral pod names rotating on every deploy compound this. Mitigation: use metric_relabel_configs to drop or replace offending labels at scrape time, aggregate paths into templated routes, move high-variance data into logs or traces instead of labels, and precompute with recording rules. Set sample and label limits per scrape to fail loudly.

COMMON WRONG ANSWERS Merely vertically scaling Prometheus memory, blaming query volume rather than series count, or suggesting more frequent scraping. Adding labels to fix it makes it worse.

LIKELY FOLLOW-UPS How do recording rules help? When would you shard or move to Thanos, Mimir, or Cortex? How do you enforce limits proactively?

ONE CONCRETE EXAMPLE A team added a user_id label to http_requests_total. With one million users, that single counter became one million time series. Active series jumped, memory ballooned, and PromQL slowed. A metric_relabel_config dropping user_id collapsed it back to a handful of series, and per-user analysis moved to traces.