Redesigning a high-cardinality request metric

WHAT THIS TESTS This evaluates whether you can define cardinality precisely and then apply that understanding to fix a specific anti-pattern metric, the practical skill that matters on the job.

A GOOD ANSWER COVERS Cardinality is the number of distinct time series a metric produces, equal to the number of unique combinations of its label values. A monitoring system like Prometheus stores one independent series per combination and holds active series in memory, so cardinality is the dominant driver of memory use, query cost, and stability. The example http_requests_total with labels user_id and path is dangerous because user_id is effectively unbounded, growing with every user, and a raw path with query strings or embedded IDs is near-unbounded too. The product of those two can reach millions or more of series, risking out-of-memory crashes and slow queries.

HOW TO REDESIGN Remove user_id from the metric entirely, since per-user breakdown does not belong in a metric label; if you need to investigate a specific user, use logs or traces keyed by user or trace ID. Normalize the path into a bounded route template, replacing /users/123/orders/456 with /users/:id/orders/:id, so distinct logical endpoints, not distinct IDs, become label values. Keep genuinely low-cardinality, useful labels such as method and status code. The result, http_requests_total with method, route, and status, has a small, bounded series count while still supporting the queries that drive alerts and dashboards.

COMMON WRONG ANSWERS Keeping user_id but reducing the scrape interval, which does not reduce series. Confusing cardinality with the number of samples. Templating the path but leaving the unbounded user_id in place.

LIKELY FOLLOW-UPS How do you detect a cardinality spike early? How do you normalize paths reliably? Where exactly should per-user analysis live? How do recording rules and relabeling help bound series?

ONE CONCRETE EXAMPLE Before, with one hundred thousand users and thousands of raw paths, the metric could approach hundreds of millions of series and crash the server. After dropping user_id and templating the path to a few dozen routes, with a few methods and a handful of status codes, the series count falls to the low thousands, fully sustainable, while you can still query error rate by route and method, and pivot to traces for any single user's experience.