Average latency is up, but p99 is flat. Why?
This tests your grasp of latency distributions. Hypothesize that a large group of typical requests slowed, pulling up the average but not crossing the p99 threshold. Segment by endpoint or customer to find the cohort.
What's really being asked
Your ability to interpret statistical metrics and translate that interpretation into a structured debugging plan. The key insight is that an increase in the mean without an increase in p99 points to a degradation in the bulk of the distribution (e.g., p50-p95), not the extreme tail. The interviewer wants to see you reason from the metrics, not just guess at causes.
The full answer
A good answer has four parts. First, state the core hypothesis: a large volume of previously fast or 'typical' requests have become slower, pulling the arithmetic mean up, but not enough to become the new p99. Second, explicitly rule out issues with the absolute slowest requests, since the p99 is flat. Third, propose a clear segmentation strategy to find the affected cohort. This involves breaking down latency data (p50, p90, average) by dimensions like API endpoint, customer ID or pricing tier, geographic region, or a new software canary group. Fourth, mention analyzing the full distribution, for example, by plotting latency histograms from before and after to visually confirm the shift in the middle of the curve.
The mistakes people make
A frequent mistake is to immediately focus the investigation on the p99 cohort, even though the prompt says its performance is unchanged. Another red flag is suggesting generic debugging steps like 'I'd check the logs' without specifying what data segments or patterns you're looking for. A candidate who confuses the properties of mean, median, and percentiles will also fail this question. For example, treating the mean as a 'typical' user experience is often false for skewed distributions like latency.
What usually comes next
Expect questions that test your ability to act on your findings. For instance, 'You discover the issue is a new, high-volume, free-tier customer. What are your recommendations?' This tests your thinking on rate limiting, capacity planning, and product trade-offs. Or, 'How would you set up an alert to catch this specific pattern in the future?' This probes your monitoring philosophy, looking for answers like creating a composite alert on the delta between the average and p95.
A concrete example
A service has a p99 latency of 800ms, driven by a complex report-generation endpoint. A different, high-volume 'get-item' endpoint typically takes 30ms. A code change adds a new check that increases the 'get-item' latency to 100ms. This 70ms increase for a huge number of requests will significantly raise the overall average. However, since 100ms is far below the 800ms p99 threshold, the p99 value remains unchanged. The problem is a degradation for a large group of previously fast requests.
Interview question
A monitoring dashboard shows that the average latency for a critical service has increased, but the p99 latency has remained unchanged. What does this pattern most strongly suggest?
- a.The system is encountering new, extremely high-latency requests that are rare enough not to affect the p99 percentile.
- b.A large proportion of previously fast or typical requests have experienced a moderate increase in their processing time.Correct
- c.The absolute slowest 1% of requests are now performing even worse, but their impact on the overall average is minimal.
- d.There's a widespread, but minor, slowdown affecting all requests equally, which is more visible in the average.
Why? this is the answer
An increase in average latency with a stable p99 indicates that the bulk of requests (e.g., p50-p95) have become slower, pulling up the mean, but not reaching the extreme latency values that define the p99. Option C is incorrect because if the slowest 1% were performing worse, the p99 would increase.
Just read this? Test yourself on what you have been reading.
Read the original → aerospike.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on analytics — each one lists the topics its interview covers.
See open roles