First steps on a p99 latency page

WHAT THIS TESTS Whether you approach an incident with a disciplined, hypothesis-driven triage process instead of randomly poking at the system.

A GOOD ANSWER COVERS First, confirm and scope the problem: verify the alert is real and not a monitoring glitch, then determine the blast radius. Is p99 elevated across all endpoints or just one, all regions or one zone, the whole fleet or a single instance, and is it actually hurting users or just one noisy path. Scope narrows the search dramatically. Second, ask what changed: recent deployments, configuration changes, feature-flag flips, schema migrations, or a sudden traffic surge are the most common triggers, so check the deploy and change timeline against when latency rose. Third, look at dependencies and saturation: is a downstream service or the database slow, are connection pools or queues backed up, are CPU, memory, or GC pauses spiking. A representative trace from a slow request quickly localizes which span is responsible.

COMMON WRONG ANSWERS Immediately restarting services or rolling back blindly before understanding scope or cause. Diving into source code first. Ignoring the change timeline, which is the highest-yield first check. Assuming the cause is in your service rather than a dependency.

LIKELY FOLLOW-UPS How would a trace narrow it down? When do you mitigate before you fully diagnose? How do you decide to roll back?

ONE CONCRETE EXAMPLE Paged on p99, you confirm it is real and see it is isolated to the search endpoint in one region. The change log shows a deploy 10 minutes earlier. A trace shows the new code making an extra synchronous call to a slow database query. You roll back the deploy to mitigate, and latency recovers, then investigate the query offline.