High cardinality in time-series databases

WHAT THIS TESTS Whether you understand that in a labeled time-series database, each unique combination of label values creates a separate stored series, and that this count, the cardinality, is the dominant cost driver.

A GOOD ANSWER COVERS Cardinality is the number of distinct time series a metric produces, which is roughly the product of the number of possible values across its labels. A metric http_requests_total with labels method and status has low cardinality. Add a user_id or full URL path label and cardinality explodes into the millions. This is a problem because each active series consumes memory in the head block, enlarges the inverted index, slows scrapes and queries, and increases storage; past a point the database OOMs or queries time out. Two management strategies: first, only use labels whose value set is bounded and enumerable, such as method, status code, region, or instance, and never put unbounded identifiers like user IDs, email addresses, request IDs, or raw paths in labels. Second, reduce variety by bucketing or normalizing values, for example collapsing /users/123 to /users/:id, mapping status codes to 2xx/4xx/5xx classes, or dropping or aggregating noisy labels via relabeling before ingestion.

COMMON WRONG ANSWERS Thinking cardinality is just the number of metrics, not label combinations. Believing more labels are always better for flexibility. Putting high-variety identifiers in labels and planning to filter later.

LIKELY FOLLOW-UPS How do you detect a cardinality spike? What is relabeling? When should that detail live in logs or traces instead of metrics?

ONE CONCRETE EXAMPLE A team adds a customer_id label to a request counter. With 2 million customers, one metric becomes 2 million series; Prometheus memory balloons and queries time out. They remove customer_id from the metric, keep only plan_tier (three values) as a label, and move per-customer detail into logs and traces, restoring stable ingestion.