Latency
47 bites tagged Latency — interview questions with model answers, and 60-second explainers.
How would you instrument and query P95 API latency by region?
This tests white-box latency instrumentation and safe cardinality for percentile aggregation. Strong answer: emit histograms by region, query P95 with histogram_quantile or a log percentile, and keep trace IDs in logs only.
How would you design an experiment measuring API latency impact on retention?
Randomize users into control and delay groups; track D7 retention and P99 latency; analyze with quantile metrics. Causal experiment design linking API latency to retention. Using observational data or mean latency only.
How would you structure a backend architecture A/B test and define metrics?
This tests causal inference rigor for infrastructure changes. A strong answer covers sticky user routing, controlling for geography and time, and paired primary metrics like P99 latency and error rate.
Monitor p99 improvement from 500ms to 200ms and side effects
Tests systems-thinking on tail-latency instrumentation and metric tradeoffs. Strong answers cover histogram metrics, distributed tracing for fan-out bottlenecks, and guarding error rates, cost.
Explain dynamic batching in inference servers and its trade-off
Dynamic batching launches when a time window or max size is met, improving throughput over static batching, but short ones wait for the slowest. Inference scheduling and the latency-vs-throughput trade-off.
How would you systematically diagnose high latency in an online inference service?
Check p90/p99 and TTFT to split queuing from compute; inspect queue depth, batch size, GPU, and benchmarks; check cache. Systems reasoning across serving stack.
Describe the difference between online and batch inference.
Online uses autoscaling APIs for millisecond-to-second latency; batch uses scheduled compute for minute-to-hour latency. Your grasp of serving patterns and infra tradeoffs.
How would you systematically debug an inference API latency breach?
This tests structured debugging across the full inference stack. A strong answer traces the request path from ingress to GPU, splits TTFT from token-generation latency, inspects queuing and batching, then applies targeted fixes.
Identify RAG latency bottlenecks and propose optimizations
This tests systems thinking across the RAG pipeline. A strong answer names four bottlenecks—embedding, search, chunking, and generation—and pairs each with caching, index tuning, and distillation. Red flag: GPU scaling without indexing fixes.
How does Rust ownership avoid Go GC's non-deterministic pauses?
Tests if you know Rust's compile-time ownership eliminates GC pauses by making deallocation deterministic at scope boundaries. A strong answer contrasts Go's STW with Rust's immediate Drop and zero-cost compile-time checks.
Why can't you t-test p99 latency, and what's a valid alternative?
Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests. Knowing percentiles are order statistics, not means. Invoking CLT to justify a t-test on p99.
Mean or median for API response times?
Tests if you know latency is skewed and outlier-sensitive. Pick median over mean because hiccups distort the mean, but note median hides tail latency. Advocate for p95, p99, and max. Red flag: defending mean as representative or ignoring tail behavior.
Average latency up 50ms but p99 flat: diagnose the discrepancy
Tests if you know mean reflects full distribution while p99 is a threshold. Strong answers hypothesize body shift like cache misses or traffic mix changes, and demand histograms and segmentation by endpoint. Red flag: blaming outliers, which would raise p99.
Why prefer median and p95 over mean for API latency?
This tests statistical intuition for skewed distributions. A strong answer notes that median captures typical experience, p95 captures tail suffering, and mean hides outliers. A red flag is claiming mean alone is sufficient.
How would you measure P95 latency by geographic region?
Tests your ability to translate a business need into a concrete observability implementation. A good answer involves instrumenting the API with a histogram metric, adding a region label via GeoIP, and querying with `histogram_quantile`.
Why can't we t-test p99 latency in an A/B test?
This tests if you know why t-tests fail for percentiles. A t-test requires a normally distributed statistic (like the mean), but a sample p99's distribution isn't normal.
Average latency is up 50ms, but p99 is flat. How do you diagnose this?
Tests your grasp of latency metrics. A rising average with flat p99 means the *bulk* of requests (p50-p90) slowed, not the tail. Hypothesize a common bottleneck and segment data by endpoint/user to find it. Red flag: blaming new, slow outliers.
Why use median/p95 for API latency instead of the mean?
This tests if you understand how long-tail distributions make averages misleading for user experience. A good answer explains that median (p50) shows the typical user, while p95 captures the worst-case experience.
Why not t-test p99 latency? Describe a valid alternative.
This tests your grasp of statistical test assumptions. A good answer explains why p99 violates t-test normality, then outlines a resampling method like bootstrapping to build a confidence interval on the *difference* of p99s.
Mean vs. Median for API Response Times?
Tests if you know latency data has outliers. A good answer chooses median because it's robust to extremes. Explain that mean gets skewed by a few slow requests (e.g., GC pauses), hiding the typical user experience.
Average latency is up, but p99 is flat. Why?
This tests your grasp of latency distributions. Hypothesize that a large group of typical requests slowed, pulling up the average but not crossing the p99 threshold. Segment by endpoint or customer to find the cohort.
Why use p50/p95 over mean for API response times?
Tests your grasp of statistical distributions for UX metrics. A good answer explains how outliers skew the mean, while percentiles (p50, p95) better represent typical and worst-case user experiences.
Serverless Cold Starts: The Price of 'Pay-per-Use'
A serverless cold start is the initial delay when a function boots from scratch, like waking a computer from being off versus asleep. It happens on the first request or after inactivity.
Get Latency bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.