How would you measure P95 latency by geographic region?

WHAT THIS TESTS: This question evaluates your practical knowledge of observability engineering. It's not just about knowing what P95 is, but how to actually produce it. Interviewers are looking for your understanding of metric types (histograms vs. summaries), the challenges of high-cardinality data, and the trade-offs between different observability signals (metrics vs. logs). Can you design a system that is both accurate and cost-effective?

A GOOD ANSWER COVERS: A strong answer walks through four distinct steps. First, instrumentation: modify the application or middleware to capture the request duration from start to finish. Second, data enrichment: add a region label to each measurement. This is typically done at the edge (CDN, load balancer) via a GeoIP lookup on the source IP address. It's critical to use broad regions (e.g., us-east-1, eu-west-1) to manage cardinality. Third, metric selection: use a histogram metric type (e.g., api_request_latency_seconds_bucket). This is the most efficient way to capture data for calculating quantiles on the server side. Fourth, querying: explain the query you'd run. In PromQL, this would be histogram_quantile(0.95, sum(rate(api_request_latency_seconds_bucket[5m])) by (le, region)).

COMMON WRONG ANSWERS: A major red flag is suggesting logging every single request's latency and region, then running a text search. This is incredibly inefficient, expensive, and slow for large-scale systems. Another mistake is using the wrong metric type. A Prometheus summary metric is problematic because its quantiles are calculated on the client side and cannot be aggregated across instances or grouped by region after the fact. A candidate who doesn't mention managing cardinality by using predefined regions instead of raw IPs or cities also shows a lack of senior-level experience.

LIKELY FOLLOW-UPS: Expect questions about trade-offs. "Why a histogram and not a summary?" (Answer: aggregatability). "What if the GeoIP lookup adds latency?" (Answer: do it asynchronously or at the edge/load balancer layer before it hits the app). "How would you handle the cost of this new high-cardinality metric?" (Answer: discuss cardinality limits, aggregation rules, or sampling). "What if we wanted P95 per customer AND region?" (Answer: this is a cardinality explosion; explain why this is a bad idea for metrics and better suited for a dedicated analytics/logging system).

ONE CONCRETE EXAMPLE: We run a global service and need to track latency for our EU users. Our load balancer does a GeoIP lookup on ingress, adding an X-User-Region: eu-west-1 header to requests. Our Go application has middleware that wraps the API handler. It starts a timer, calls the handler, then records the duration in a Prometheus histogram metric: api_requests_latency_seconds.WithLabelValues("eu-west-1", "/v1/critical_endpoint").Observe(duration). To get the report, we run histogram_quantile(0.95, sum(rate(api_requests_latency_seconds_bucket{endpoint="/v1/critical_endpoint"}[5m])) by (le, region)) in our Grafana dashboard.