Average latency is up 50ms, but p99 is flat. How do you diagnose this?

What's really being asked

This question tests your analytical skills and deep understanding of performance metrics. The interviewer wants to see if you can correctly interpret the statistical signals. A rising average (mean) with a flat p99 indicates that the main body of the latency distribution has shifted slower, while the extreme tail has not. The problem is a widespread, moderate slowdown affecting a large number of requests, not a few new, very slow ones.

The full answer

A strong answer moves from insight to a specific diagnostic plan. First, state the core insight: the average is being pulled up by a slowdown in the majority of requests (the p50-p90 range), while the p99, which tracks the slowest 1%, is unaffected. Second, form hypotheses based on this. The cause must be in a common code path or resource. Good hypotheses include: a recent code deploy added a fixed 50ms overhead to most requests (e.g., a new security check); a core database table used by most API calls got slightly larger, slowing down index scans; a load balancer configuration change added a small delay for all traffic. Third, outline a data segmentation plan. To find the source, you must slice the latency data. Key segments to analyze are: by API endpoint, by customer tier, by server/region, or by software version. Fourth, specify the metrics for these segments. You would compare the p50 and p75 latencies for each segment before and after the change. A jump in the median for a specific endpoint would be a smoking gun.

The mistakes people make

The biggest red flag is misinterpreting the signal and focusing on the tail of the distribution. A candidate might suggest looking for new, slow outliers, but that would have increased the p99. Vague answers like "I'd check the logs" or "I'd look at CPU utilization" are weak. A senior answer is specific: "I'd segment latency by API endpoint and compare their p50s before and after the incident, hypothesizing that a high-traffic endpoint is the source." Blaming "the network" without a plan to prove it is also a poor response.

What usually comes next

Expect the interviewer to give you a result and ask "What next?". For example: "You find the slowdown is isolated to your '/api/v2/users' endpoint. Now what?" (Answer: Isolate recent code changes to that endpoint's controller, check the performance of its specific database queries, and analyze its downstream service calls). Or, "The slowdown only affects your free-tier customers. What's your hypothesis?" (Answer: A noisy neighbor problem on shared infrastructure, or a change in traffic patterns from a single large free user causing resource contention).

A concrete example

Imagine before the change, 99% of requests take 100ms and the slowest 1% take 800ms. The average latency is (0.99 100ms) + (0.01 800ms) = 107ms. The p99 latency is 800ms. Now, a change adds 50ms of latency to the fastest 99% of requests, but the slowest 1% are unaffected. The new average is (0.99 150ms) + (0.01 800ms) = 156.5ms, an increase of about 50ms. The p99 latency remains 800ms. This perfectly matches the scenario.