How would you measure P95 latency by geographic region?

WHAT THIS TESTS: This question tests your ability to translate a business requirement into a concrete technical implementation for observability. It specifically probes your understanding of white-box monitoring, the trade-offs between metrics and logs, the correct use of advanced metric types like histograms for calculating percentiles, and your familiarity with a query language like PromQL. Interviewers are looking for a cost-effective, scalable, and accurate solution.

A GOOD ANSWER COVERS: An excellent answer addresses four points in order. First, clarify the data source for the region, typically by performing a GeoIP lookup on the source IP address at the infrastructure edge (like a load balancer or CDN) and passing it to the application as a request header. Second, describe the instrumentation within the application code. You would use a metrics library to capture the request duration and record it in a Histogram metric type. Third, explain the importance of labeling. The metric should be labeled with the endpoint and, crucially, the region, e.g., api_request_duration_seconds{endpoint="/v1/data", region="eu-west-1"}. Fourth, provide the query. For Prometheus, the PromQL query would be histogram_quantile(0.95, sum(rate(api_request_duration_seconds_bucket[5m])) by (le, region)). You should be able to briefly explain that rate() handles counter resets, sum() aggregates across all application instances, and histogram_quantile() performs the final calculation.

COMMON WRONG ANSWERS: One major red flag is suggesting a logs-only solution. While logging every request with its latency and region works, it's far more expensive to store and query than a pre-aggregated metric for this use case. A senior answer acknowledges logs are better for debugging individual requests, not for dashboarding percentiles. Another common mistake is suggesting the use of an average instead of a percentile, which hides the tail latency that P95 is designed to reveal. Finally, using the wrong metric type is a frequent error. A Prometheus Summary metric calculates quantiles client-side, making them impossible to aggregate correctly across multiple instances, which is why Histograms are the correct choice for service-wide percentile calculations.

LIKELY FOLLOW-UPS: Expect follow-ups like: "What if we also wanted to see P99.9? What are the limitations of your histogram?" (Answer: Accuracy depends on bucket configuration; you need smaller, well-placed buckets at the high end). Or, "Your service runs on 200 pods. How does your query handle that?" (Answer: The sum(...) by (...) clause in the PromQL query correctly aggregates the data from all pods before calculating the quantile). Or, "What is the performance overhead of this instrumentation?" (Answer: Minimal; client libraries are highly optimized, and a single metric update per request is negligible, typically microseconds).

ONE CONCRETE EXAMPLE: If our P95 SLO for this endpoint is 250ms, our histogram buckets must be configured to provide good resolution around this value. We might define buckets in seconds like: [..., 0.1, 0.2, 0.25, 0.3, 0.5, 1.0]. If a query for us-east-1 returns a P95 of 480ms, we know that 5% of users in that region are experiencing latencies of nearly half a second or more, clearly violating our SLO. The precision of that 480ms figure is determined by the spacing of the 0.3 and 0.5 buckets.