How would you measure P95 latency by geographic region?

What's really being asked

This question tests your ability to translate a business requirement into a concrete technical implementation for observability. It specifically probes your understanding of white-box monitoring, the trade-offs between metrics and logs, the correct use of advanced metric types like histograms for calculating percentiles, and your familiarity with a query language like PromQL. Interviewers are looking for a cost-effective, scalable, and accurate solution.

The full answer

An excellent answer addresses four points in order. First, clarify the data source for the region, typically by performing a GeoIP lookup on the source IP address at the infrastructure edge (like a load balancer or CDN) and passing it to the application as a request header. Second, describe the instrumentation within the application code. You would use a metrics library to capture the request duration and record it in a Histogram metric type. Third, explain the importance of labeling. The metric should be labeled with the endpoint and, crucially, the region, e.g., api_request_duration_seconds{endpoint="/v1/data", region="eu-west-1"}. Fourth, provide the query. For Prometheus, the PromQL query would be histogram_quantile(0.95, sum(rate(api_request_duration_seconds_bucket[5m])) by (le, region)). You should be able to briefly explain that rate() handles counter resets, sum() aggregates across all application instances, and histogram_quantile() performs the final calculation.

The mistakes people make

One major red flag is suggesting a logs-only solution. While logging every request with its latency and region works, it's far more expensive to store and query than a pre-aggregated metric for this use case. A senior answer acknowledges logs are better for debugging individual requests, not for dashboarding percentiles. Another common mistake is suggesting the use of an average instead of a percentile, which hides the tail latency that P95 is designed to reveal. Finally, using the wrong metric type is a frequent error. A Prometheus Summary metric calculates quantiles client-side, making them impossible to aggregate correctly across multiple instances, which is why Histograms are the correct choice for service-wide percentile calculations.

What usually comes next

Expect follow-ups like: "What if we also wanted to see P99.9? What are the limitations of your histogram?" (Answer: Accuracy depends on bucket configuration; you need smaller, well-placed buckets at the high end). Or, "Your service runs on 200 pods. How does your query handle that?" (Answer: The sum(...) by (...) clause in the PromQL query correctly aggregates the data from all pods before calculating the quantile). Or, "What is the performance overhead of this instrumentation?" (Answer: Minimal; client libraries are highly optimized, and a single metric update per request is negligible, typically microseconds).

A concrete example

If our P95 SLO for this endpoint is 250ms, our histogram buckets must be configured to provide good resolution around this value. We might define buckets in seconds like: [..., 0.1, 0.2, 0.25, 0.3, 0.5, 1.0]. If a query for us-east-1 returns a P95 of 480ms, we know that 5% of users in that region are experiencing latencies of nearly half a second or more, clearly violating our SLO. The precision of that 480ms figure is determined by the spacing of the 0.3 and 0.5 buckets.