How would you systematically debug an inference API latency breach?

WHAT THIS TESTS: Whether you can treat latency debugging as a structured systems problem rather than a checklist. Interviewers want to see that you understand the difference between time-to-first-token and inter-token latency, that you know how requests flow through load balancers, service meshes, Kubernetes schedulers, and GPU kernels, and that you prioritize measurement over assumptions.

A GOOD ANSWER COVERS: First, instrumentation and segmentation. You would deploy distributed tracing across the ingress gateway, API server, inference engine, and model runtime, capturing p90 and p95 percentiles rather than averages because tail latency drives SLO violations. Second, lifecycle decomposition. Break the path into network ingress and TLS termination, load balancer queuing, pod scheduling and cold-start penalties, request batching logic, GPU kernel execution, and memory-bound operations like KV-cache updates. Third, bottleneck identification. If TTFT is high, look at queuing depth, batch size, or model weight loading. If per-token latency is high, look at GPU memory bandwidth, tensor parallelism efficiency, or suboptimal batching. Fourth, targeted remediation. Right-size dynamic batching, enable quantization or KV-cache-aware routing, tune concurrency limits so you do not overcommit GPU streams, and use topology-aware scheduling to reduce cross-node traffic. Scale replicas only after proving the bottleneck is compute saturation rather than network or scheduling.

COMMON WRONG ANSWERS: Jumping immediately to use a smaller model or add more GPUs without profiling. Treating throughput and latency as interchangeable metrics. Ignoring Kubernetes scheduling delays or HPA cold starts. Suggesting average latency is sufficient for SLO tracking instead of p95 or p99. Proposing batching increases without acknowledging that larger batches can hurt TTFT.

LIKELY FOLLOW-UPS: How would you decide between tensor parallelism and pipeline parallelism for a large model? What metrics would you expose to distinguish queuing latency from GPU execution time? How do you handle latency spikes during rolling updates or model version swaps? When is quantization unacceptable because it harms model accuracy?

ONE CONCRETE EXAMPLE: Suppose p95 TTFT is 800 milliseconds against a 500 millisecond SLO. Tracing shows 50 milliseconds in the ingress gateway, 200 milliseconds in a pod pending queue because the HPA target was CPU-based instead of GPU-queue-depth-based, and 550 milliseconds in model execution. You fix the HPA to scale on custom GPU queue metrics, dropping queue time to 20 milliseconds. You then reduce batch max tokens from 256 to 128, which cuts execution to 300 milliseconds but lowers throughput slightly. Total p95 TTFT falls to 370 milliseconds, restoring the SLO without changing the model architecture.