Average latency up 50ms but p99 flat: diagnose the discrepancy

What's really being asked

This question tests statistical literacy for latency distributions. The interviewer wants to know if you grasp that the arithmetic mean is sensitive to every request while the p99 is merely a threshold that 99 percent of requests must stay under. It also tests diagnostic rigor: do you reach for histograms and segmentation instead of staring at aggregate percentiles.

The full answer

First, form hypotheses that place the slowdown in the body of the distribution, between roughly the 50th and 98th percentiles. Common culprits are a drop in cache hit rate that turns fast hits into moderately slow misses, a traffic mix shift toward a heavier endpoint, a new client sending larger payloads, or a code change that added a fixed 50 ms to a hot path. Second, explain that because p99 remained flat, the slowest 1 percent did not get worse, so the problem is not tail events like garbage collection pauses or disk stalls. Third, describe the exact data segments you would analyze: request latency histograms or CDFs to visualize where the mass moved; breakdown by API endpoint, service version, and deployment region; decomposition into network queue time versus processing time; and filtering by cache hit or miss, payload size bucket, and status code. Fourth, mention checking for bimodality or a new subpopulation that is consistently slower than the old median but faster than the old p99.

The mistakes people make

The biggest red flag is blaming outliers. Outliers raise the maximum or deep tail percentiles like p99 or p99.9; they do not uniquely inflate the mean while leaving p99 untouched. Another red flag is suggesting that you need more samples or that the discrepancy is just noise; a 50 ms shift in mean with flat p99 is a real signal about the distribution shape. A third red flag is proposing to optimize the tail when the tail did not move.

What usually comes next

The interviewer may ask how you would confirm your hypothesis with a single histogram, what you would do if p50 were also flat while the mean rose, or how a change in request volume from a specific region could create this pattern. They might also ask you to sketch the before and after CDF curves.

A concrete example

Imagine a search API where 80 percent of queries were cached and returned in 10 ms, while 20 percent were uncached and returned in 200 ms, producing a mean of 48 ms and a p99 of 200 ms. If the cache hit rate drops to 60 percent, the new mean climbs to 86 ms, but the p99 stays at 200 ms because the uncached tail has not changed. Segmenting by cache status immediately reveals the culprit.