How would you systematically diagnose high latency in an online inference service?

WHAT THIS TESTS: Structured end-to-end debugging instincts across the full serving stack. Interviewers want to see whether you separate queuing delays from compute delays, whether you know that tail latency drives user perception, and whether you can move from symptoms to root cause without guessing.

A GOOD ANSWER COVERS: A strong response moves through layers in order. First, establish the latency baseline and focus on tail percentiles rather than averages because a healthy mean can hide a p99 of twelve seconds. Measure Time to First Token separately from total generation time, especially for streaming endpoints. Second, inspect infrastructure signals: queue depth, actual versus configured batch size, GPU utilization, and cold-start frequency. Third, isolate the model itself with a benchmark using a tool like trtexec to determine the theoretical minimum inference time. Fourth, examine the application layer for cache-aware routing, serialization overhead, and request routing logic. Fifth, if autoscaling is involved, check whether latency spikes correlate with scale-from-zero events and discuss mitigations such as keep-alives or adapter size tuning.

COMMON WRONG ANSWERS: The biggest red flag is jumping straight to model optimization, quantization, or architecture changes without first checking queue saturation or batching misconfiguration. Another weak pattern is citing average latency as the primary metric; interviewers expect you to call out p90, p95, and p99. A third mistake is ignoring TTFT and treating all latency as total generation time, which misses the user-perceived lag in streaming applications.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would verify that a fix actually worked in production without regressing throughput. They may also ask you to compare continuous batching versus static batching, or to design a dashboard that would have caught the issue within five minutes. You might also be asked to estimate cost trade-offs between keeping instances warm and accepting occasional cold-start latency.

ONE CONCRETE EXAMPLE: Suppose your p99 jumps from two seconds to eight seconds during peak traffic while GPU utilization stays flat at forty percent. Flat GPU with rising latency strongly suggests a queuing bottleneck rather than a model compute issue. You check queue depth and see it climb from five to two hundred requests. The root cause is a configured max batch size of one with autoscaling that scales too slowly; instances spend their time cold-starting instead of processing. The fix is to enable continuous batching, raise the max batch size to sixteen, and add keep-alive pings to prevent scale-to-zero, which brings p99 back under three seconds.