How would you instrument and query P95 API latency by region?

What's really being asked

This question evaluates your ability to implement white-box monitoring for a critical service latency metric while managing label cardinality. The interviewer wants to see that you understand the difference between metrics and logs, know how to aggregate percentiles from histograms or timers, and can avoid operational anti-patterns that crash a time-series database.

The full answer

First, instrument the application code to record request durations as histograms or timers with a region label. In Prometheus this means using a Histogram with predefined buckets and a label like region with values such as us-east or eu-west. In a log pipeline you would emit a structured log line containing duration_ms and region on every request. Second, explain the query. In Prometheus you use histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (region, le)). In a log tool you group by region and calculate the 95th percentile of duration_ms over the window. Third, address cardinality safety. Region is low cardinality, usually under one hundred values, so it is safe for a metric label. High cardinality fields like trace_id or user_id must stay in logs. Fourth, mention that you would expose this on a dashboard for trend analysis and keep raw logs for drill-down.

The mistakes people make

A major red flag is suggesting to store every raw latency value as a gauge or to put unbounded strings like request paths or user IDs into Prometheus labels. This causes cardinality explosion and can overwhelm the metrics backend. Another mistake is trying to average percentiles or pre-compute a global P95 on the client side without keeping per-region breakdowns. Some candidates also forget to specify the time window or rate interval in the query, which makes the result meaningless.

What usually comes next

The interviewer might ask how you would handle a region with very low traffic where a single outlier skews the P95. They might also ask how to reduce storage cost if you have thousands of microservices emitting histograms, or how you would correlate a latency spike with specific trace IDs. Be ready to discuss log-to-metric pipelines and exemplars.

A concrete example

Suppose your API runs in three regions and you define a Prometheus histogram with buckets at 10ms, 50ms, 100ms, 250ms, 500ms, and 1s. You label each observation with region. To generate the weekly report for the product manager, you run histogram_quantile(0.95, sum(rate(api_request_duration_seconds_bucket[1h])) by (region, le)) over the past seven days and export the results to a dashboard table. If eu-west shows 400ms P95 while us-east shows 80ms, you filter your structured logs to eu-west and trace_id to find the specific slow endpoints.