Cardinality: The Hidden Cost of Time-Series Metrics

WHY IT EXISTS Monitoring systems need to slice and dice data using labels for powerful queries. But this flexibility creates a new problem: tracking too many unique combinations can overwhelm the system. Cardinality management is the discipline of balancing this observability power with system stability.

THE MENTAL MODEL Think of cardinality like a spreadsheet's rows. Each unique time series is a row that needs to be indexed and stored. A metric http_requests with a label status having 3 values creates 3 rows. Add a path label with 100 values, and you now have 3 * 100 = 300 rows. Add a user_id label with 1 million values, and you have 300 million rows. Your spreadsheet, and your time-series database, will crash.

HOW IT WORKS A time series is uniquely identified by its name and its set of key-value labels. For example, api_calls{method="GET", path="/users"} is one series, while api_calls{method="POST", path="/users"} is another. The total number of these unique combinations is the system's cardinality. Time-series databases build an in-memory index of these combinations for fast lookups. High cardinality bloats this index, consuming RAM and slowing down queries that must sift through millions of potential series.

WHEN TO USE IT This isn't a technique to "use," but a concept to manage. Always be mindful of cardinality when designing metrics. Use labels for dimensions with a small, bounded set of values. Good examples include HTTP method (GET, POST), status code groups (2xx, 4xx), or environment (prod, staging). These allow for powerful, efficient aggregation and alerting without overwhelming the system.

WHEN NOT TO USE IT Never use labels for values with unbounded or extremely high cardinality. This is the primary footgun. Avoid putting identifiers like user IDs, email addresses, session IDs, request IDs, or full URL paths with variables into metric labels. This high-cardinality information belongs in logs or distributed tracing systems, which are designed for event data, not aggregated metrics.

ONE CANONICAL EXAMPLE A team adds a customer_id label to a Prometheus metric http_requests_total to debug issues for specific customers. The system has 2 million customers. Even if only 10,000 are active daily, this creates 10,000 new time series. Combined with other labels like path (50 values) and status (5 values), this one change adds 10,000 * 50 * 5 = 2,500,000 new series to Prometheus. The server's memory usage skyrockets, queries time out, and the monitoring system becomes unusable.