How would you measure P95 latency by geographic region?
Tests your ability to translate a business need into a concrete observability implementation. A good answer involves instrumenting the API with a histogram metric, adding a region label via GeoIP, and querying with histogram_quantile.
What's really being asked
This question evaluates your practical knowledge of observability engineering. It's not just about knowing what P95 is, but how to actually produce it. Interviewers are looking for your understanding of metric types (histograms vs. summaries), the challenges of high-cardinality data, and the trade-offs between different observability signals (metrics vs. logs). Can you design a system that is both accurate and cost-effective?
The full answer
A strong answer walks through four distinct steps. First, instrumentation: modify the application or middleware to capture the request duration from start to finish. Second, data enrichment: add a region label to each measurement. This is typically done at the edge (CDN, load balancer) via a GeoIP lookup on the source IP address. It's critical to use broad regions (e.g., us-east-1, eu-west-1) to manage cardinality. Third, metric selection: use a histogram metric type (e.g., api_request_latency_seconds_bucket). This is the most efficient way to capture data for calculating quantiles on the server side. Fourth, querying: explain the query you'd run. In PromQL, this would be histogram_quantile(0.95, sum(rate(api_request_latency_seconds_bucket[5m])) by (le, region)).
The mistakes people make
A major red flag is suggesting logging every single request's latency and region, then running a text search. This is incredibly inefficient, expensive, and slow for large-scale systems. Another mistake is using the wrong metric type. A Prometheus summary metric is problematic because its quantiles are calculated on the client side and cannot be aggregated across instances or grouped by region after the fact. A candidate who doesn't mention managing cardinality by using predefined regions instead of raw IPs or cities also shows a lack of senior-level experience.
What usually comes next
Expect questions about trade-offs. "Why a histogram and not a summary?" (Answer: aggregatability). "What if the GeoIP lookup adds latency?" (Answer: do it asynchronously or at the edge/load balancer layer before it hits the app). "How would you handle the cost of this new high-cardinality metric?" (Answer: discuss cardinality limits, aggregation rules, or sampling). "What if we wanted P95 per customer AND region?" (Answer: this is a cardinality explosion; explain why this is a bad idea for metrics and better suited for a dedicated analytics/logging system).
A concrete example
We run a global service and need to track latency for our EU users. Our load balancer does a GeoIP lookup on ingress, adding an X-User-Region: eu-west-1 header to requests. Our Go application has middleware that wraps the API handler. It starts a timer, calls the handler, then records the duration in a Prometheus histogram metric: api_requests_latency_seconds.WithLabelValues("eu-west-1", "/v1/critical_endpoint").Observe(duration). To get the report, we run histogram_quantile(0.95, sum(rate(api_requests_latency_seconds_bucket{endpoint="/v1/critical_endpoint"}[5m])) by (le, region)) in our Grafana dashboard.
Interview question
To calculate P95 latency aggregated by region across a fleet of servers, which metric type is the most suitable choice?
- a.A gauge, to store the most recent request latency for each region.
- b.A summary, as it efficiently calculates quantiles on each application instance before collection.
- c.A histogram, because its bucketed observations can be aggregated across all instances before calculating quantiles.Correct
- d.A counter, to count requests slower than a specific threshold for each region.
Why? this is the answer
A histogram is correct because its raw bucket data can be aggregated server-side, allowing you to calculate quantiles across dimensions like 'region'. A summary is a tempting but incorrect choice because its quantiles are calculated client-side and cannot be correctly aggregated.
Just read this? Test yourself on what you have been reading.
Read the original → sre.google
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on observability — each one lists the topics its interview covers.
See open roles