Top 30 Latency Interview Questions and Answers

30 questions. Pick an answer, or open “Show the answer” to read it.

Answers are graded in your browser. Nothing is saved, and no XP or streak is earned here. The app keeps score.

1. Question 1 of 30

   Which set correctly lists the Four Golden Signals for monitoring a user-facing system?

   • aCPU, memory, disk, and network utilization
   • bLatency, traffic, errors, and saturation
   • cAvailability, durability, throughput, and cost
   • dLogs, metrics, traces, and dashboards
   Show the answer

   Answer: b · Latency, traffic, errors, and saturation

   The Four Golden Signals are latency, traffic, errors, and saturation, focused on user experience and capacity. CPU and memory are resource metrics; logs and traces are observability pillars, not the golden signals.

   Read the full bite: What are the Four Golden Signals?

2. Question 2 of 30

   When is alerting on p99 latency rather than p95 most justified?

   • aAlways, because p99 is strictly more accurate than any other percentile
   • bWhen the average latency is already below the target threshold
   • cWhen requests fan out to many backends so the slow tail dominates user-perceived latency
   • dWhen you want a single number that summarizes typical user experience
   Show the answer

   Answer: c · When requests fan out to many backends so the slow tail dominates user-perceived latency

   High fan-out makes a backend's tail the common case for users, so p99 matters most there. p99 is not universally better, the median describes typical experience, and a healthy average can still hide a bad tail.

   Read the full bite: Why use latency percentiles over the average?

3. Question 3 of 30

   When A/B testing a backend architecture change with shared database pools, which experimental design most rigorously isolates the architecture effect from confounders?

   • aDeploy a 5% canary of the new variant, measure P99 latency, and compare against historical data from the week before
   • bBucket users by session ID hash for sticky routing, stratify by region, pair P99 latency with a user completion metric, and run a parallel holdback at identical traffic percentage
   • cSplit requests 50/50 without stickiness, use mean latency as the primary metric, and compare against pre-launch baseline data
   • dHash user IDs for sticky routing and track P99 latency, but skip stratification and rely only on server-side error rate without a user guardrail
   Show the answer

   Answer: b · Bucket users by session ID hash for sticky routing, stratify by region, pair P99 latency with a user completion metric, and run a parallel holdback at identical traffic percentage

   Option B correctly combines sticky routing, regional stratification, paired P99 and user-completion metrics, and a parallel holdback to isolate architecture effects. Option D is tempting because it includes stickiness and P99 latency, but omitting stratification and a user-facing guardrail leaves the test vulnerable to regional confounders and silent UX regressions.

   Read the full bite: How would you structure a backend architecture A/B test and define metrics?

4. Question 4 of 30

   For an auth API availability SLI, why should legitimate 401 responses for wrong passwords be excluded from the failure count?

   • aBecause they represent the system working correctly, not an outage, so counting them penalizes correct behavior
   • bBecause 401 responses are always caused by network errors
   • cBecause client errors are impossible to measure at the edge
   • dBecause only 5xx codes can be measured by load balancers
   Show the answer

   Answer: a · Because they represent the system working correctly, not an outage, so counting them penalizes correct behavior

   A 401 for a wrong password is the auth system doing its job, so treating it as downtime would distort the SLI and punish correct behavior. Server-side 5xx and timeouts are the real availability failures to count.

   Read the full bite: Proposing availability and latency SLIs for an auth API

5. Question 5 of 30

   Why can a breaching p99 with a healthy p50 cause outsized user pain in a microservice architecture?

   • aBecause the median is always misreported under load
   • bBecause the average latency is mathematically wrong
   • cBecause fan-out means one user action hits many backends, raising the chance at least one lands in the slow tail
   • dBecause p99 only measures failed requests, not slow ones
   Show the answer

   Answer: c · Because fan-out means one user action hits many backends, raising the chance at least one lands in the slow tail

   With fan-out, a single action triggers many backend calls, so the odds that at least one hits the slow one percent grow quickly, inflating the latency users actually feel. The median being fine does not protect against this multiplicative tail effect.

   Read the full bite: Diagnosing a healthy p50 but breaching p99

6. Question 6 of 30

   When setting SLOs for API latency, why can the mean alone give a misleading view of user experience?

   • aA few extreme tail values can pull the mean up and hide suffering at the p95
   • bThe mean and standard deviation fully describe latency because request times are normally distributed
   • cOutliers are usually measurement noise, so the mean smooths them away to show true performance
   • dPercentiles are only useful for large-scale systems and are too noisy for smaller services
   Show the answer

   Answer: a · A few extreme tail values can pull the mean up and hide suffering at the p95

   Latency data is right-skewed, so a handful of multi-second outliers can inflate the mean and mask terrible tail experiences. Distractor B is tempting but wrong because standard deviation assumes a normal distribution, whereas latency distributions have a long tail and a hard floor near zero.

   Read the full bite: Why prefer median and p95 over mean for API latency?

7. Question 7 of 30

   An API's latency dashboard shows a mean of 500ms but a median of 150ms. What does this discrepancy most likely indicate?

   • aThe median is misleading because it ignores outliers; the true typical experience is closer to the 500ms mean.
   • bThe system is inconsistent, with roughly half of requests taking 150ms and the other half taking 500ms.
   • cMost requests complete around 150ms, but a few very slow requests are skewing the mean upward.
   • dThe latency follows a normal (bell-curve) distribution, so this difference points to a data collection error.
   Show the answer

   Answer: c · Most requests complete around 150ms, but a few very slow requests are skewing the mean upward.

   The median represents the typical experience. A mean that is much higher than the median indicates a right-skewed distribution with a 'long tail' of a few very slow requests, which pull the average up without affecting the midpoint.

   Read the full bite: Why use median/p95 for API latency instead of the mean?

8. Question 8 of 30

   When analyzing API response times, why are percentiles (e.g., p50, p95) generally considered more informative than the arithmetic mean?

   • aPercentiles are easier for non-technical stakeholders to understand and interpret.
   • bThe mean is primarily useful for metrics that follow a normal distribution, which API latencies typically do not.
   • cPercentiles better represent the typical and worst-case user experiences because the mean can be heavily skewed by a small number of very slow requests.
   • dPercentiles offer a more precise measure of the absolute fastest and slowest requests.
   Show the answer

   Answer: c · Percentiles better represent the typical and worst-case user experiences because the mean can be heavily skewed by a small number of very slow requests.

   The card emphasizes that API response times often have a long-tail distribution where a few slow requests can dramatically skew the mean, making it a poor representation of most users' experiences. Percentiles like p50 (median) accurately reflect the typical user's experience, while p95/p99 capture the worst-case for the majority, directly linking to user satisfaction. Option B, while true that mean is better for normal distributions, doesn't fully capture the user-centric reason for preferring percentiles for latency, which is the core of the card's argument.

   Read the full bite: Why use p50/p95 over mean for API response times?

9. Question 9 of 30

   Which label is safe to add to a request latency metric without risking a cardinality explosion?

   • aThe authenticated user_id
   • bA unique request_id per call
   • cA normalized route template such as /orders/:id
   • dThe full request URL including query string
   Show the answer

   Answer: c · A normalized route template such as /orders/:id

   A normalized route template has a small, bounded set of values, keeping series counts manageable while enabling per-endpoint analysis. user_id, full URLs, and request_id are unbounded and would explode cardinality, so they belong in logs or traces.

   Read the full bite: Essential tags for a request latency metric

10. Question 10 of 30

    Why can Rust reclaim heap memory at a specific moment without the latency spikes typical of Go's GC?

    • aRust uses a reference-counting scheme for heap objects, spreading memory management cost across pointer operations.
    • bRust enforces ownership at compile time, so Drop deallocates heap memory deterministically when values go out of scope.
    • cRust's runtime schedules background sweeper threads that reclaim memory during idle CPU cycles without stopping the world.
    • dRust performs all heap allocation on the stack through compile-time size analysis, eliminating dynamic memory entirely.
    Show the answer

    Answer: b · Rust enforces ownership at compile time, so Drop deallocates heap memory deterministically when values go out of scope.

    Rust's compiler tracks ownership and lifetimes, guaranteeing that heap memory is freed immediately when a value goes out of scope via Drop, eliminating non-deterministic pauses. Distractor D confuses Rust's default compile-time ownership with opt-in reference counting types like Rc and Arc, which do incur runtime overhead but are not the standard mechanism.

    Read the full bite: How does Rust ownership avoid Go GC's non-deterministic pauses?

11. Question 11 of 30

    Why can a service like Global Accelerator speed up POST-heavy traffic that a standard CDN cache cannot?

    • aIt rewrites POST requests into cacheable GET requests automatically
    • bIt caches POST responses at the edge for later reuse
    • cIt compresses the request body so less data crosses the internet
    • dIt terminates connections at a nearby edge and routes over an optimized backbone
    Show the answer

    Answer: d · It terminates connections at a nearby edge and routes over an optimized backbone

    Acceleration shortens handshake round trips at a nearby edge and uses the provider's optimized backbone, helping uncacheable traffic. POSTs are not cacheable, so caching them is not an option.

    Read the full bite: Accelerating uncacheable dynamic traffic globally

12. Question 12 of 30

    Why are summary metrics problematic for computing a fleet-wide p99 latency across many servers?

    • aPer-instance quantiles cannot be mathematically aggregated into a global quantile
    • bSummaries only support p50 and never higher percentiles
    • cSummaries store raw observations that are too large to transfer
    • dSummaries recompute quantiles at query time, which is too slow
    Show the answer

    Answer: a · Per-instance quantiles cannot be mathematically aggregated into a global quantile

    Summary quantiles are computed per instance and are non-aggregatable, so combining them gives an invalid global figure. Histograms store additive bucket counts and compute the quantile at query time, which does aggregate correctly.

    Read the full bite: Histograms versus summaries for latency

13. Question 13 of 30

    In a production RAG pipeline, which optimization best demonstrates systems-level thinking about retrieval latency?

    • aAdd GPU nodes to the LLM inference tier and enable response caching
    • bSwitch to fixed-size chunking without overlap to simplify preprocessing
    • cRetrieve larger chunks to reduce database round-trips and network overhead
    • dTune HNSW index parameters and implement hybrid dense-plus-sparse retrieval with BM25 pruning
    Show the answer

    Answer: d · Tune HNSW index parameters and implement hybrid dense-plus-sparse retrieval with BM25 pruning

    Tuning HNSW and adding BM25 hybrid pruning directly addresses vector search as a tunable distributed component rather than a black box. Option A is tempting because scaling GPUs is a common reflex, but it ignores that retrieval and embedding can consume 30 to 50 percent of total latency while failing to address index configuration or chunking strategy.

    Read the full bite: Identify RAG latency bottlenecks and propose optimizations

14. Question 14 of 30

    For measuring p99 latency aggregated across a 30-server fleet, why are Prometheus histograms preferred over summaries?

    • aHistograms compute exact quantiles with no approximation
    • bHistogram buckets are additive so quantiles can be computed fleet-wide
    • cSummaries require more disk than histograms in every case
    • dSummaries cannot record request latency at all
    Show the answer

    Answer: b · Histogram buckets are additive so quantiles can be computed fleet-wide

    Histogram bucket counts sum across instances, so a correct fleet-wide quantile is computed at query time. Summary quantiles are per-instance and non-aggregatable; histograms are approximate, not exact.

    Read the full bite: Prometheus histogram versus summary

15. Question 15 of 30

    Why must you wrap the bucket counter in rate() and use sum by (le) before applying histogram_quantile for p95 latency?

    • aTo change the quantile from 0.95 to 0.999
    • bTo handle counter resets and aggregate buckets correctly across instances
    • cTo convert seconds into milliseconds automatically
    • dTo remove 5xx responses from the latency calculation
    Show the answer

    Answer: b · To handle counter resets and aggregate buckets correctly across instances

    rate() computes per-second increase and survives counter resets, while sum by (le) aggregates bucket counts across instances so the quantile is correct fleet-wide. It neither rescales units nor alters which quantile is requested.

    Read the full bite: Writing SLIs in PromQL

16. Question 16 of 30

    A monitoring dashboard shows that the average latency for a critical service has increased, but the p99 latency has remained unchanged. What does this pattern most strongly suggest?

    • aThe system is encountering new, extremely high-latency requests that are rare enough not to affect the p99 percentile.
    • bA large proportion of previously fast or typical requests have experienced a moderate increase in their processing time.
    • cThe absolute slowest 1% of requests are now performing even worse, but their impact on the overall average is minimal.
    • dThere's a widespread, but minor, slowdown affecting all requests equally, which is more visible in the average.
    Show the answer

    Answer: b · A large proportion of previously fast or typical requests have experienced a moderate increase in their processing time.

    An increase in average latency with a stable p99 indicates that the bulk of requests (e.g., p50-p95) have become slower, pulling up the mean, but not reaching the extreme latency values that define the p99. Option C is incorrect because if the slowest 1% were performing worse, the p99 would increase.

    Read the full bite: Average latency is up, but p99 is flat. Why?

17. Question 17 of 30

    If mean latency rises 50 ms while p99 stays flat, which diagnostic step best addresses the discrepancy?

    • aIncrease the request sample size to rule out statistical noise
    • bFocus optimization efforts on the slowest 1% of requests
    • cInspect latency histograms and segment by endpoint and cache status
    • dInvestigate garbage collection pauses and disk stalls for new outliers
    Show the answer

    Answer: c · Inspect latency histograms and segment by endpoint and cache status

    A flat p99 means the tail did not worsen, so the increase must come from the body of the distribution; histograms and segmentation by endpoint or cache status reveal where the mass shifted. Investigating outliers or tail events is incorrect because those would raise p99, not just the mean.

    Read the full bite: Average latency up 50ms but p99 flat: diagnose the discrepancy

18. Question 18 of 30

    Your service's average latency increased by 50ms, but p99 latency is unchanged. What is the most effective initial diagnostic step?

    • aAnalyze the slowest 1% of requests to identify new, extremely slow outliers.
    • bIsolate and analyze requests that are now timing out.
    • cCompare the median (p50) latency for each API endpoint before and after the increase.
    • dCheck for alerts related to server CPU or memory saturation across the fleet.
    Show the answer

    Answer: c · Compare the median (p50) latency for each API endpoint before and after the increase.

    This is correct because a flat p99 indicates the issue is not with extreme outliers but a widespread slowdown affecting the bulk of requests. Comparing median latency by endpoint helps isolate this common-path problem. Analyzing the slowest 1% is incorrect because the p99 metric shows the tail's performance has not changed.

    Read the full bite: Average latency is up 50ms, but p99 is flat. How do you diagnose this?

19. Question 19 of 30

    Which first check usually gives the highest signal when investigating a sudden p99 latency spike?

    • aThe total uptime of the affected hosts
    • bWhether a recent deploy or config change preceded the spike
    • cThe color theme of the alerting dashboard
    • dRewriting the slowest function in the service
    Show the answer

    Answer: b · Whether a recent deploy or config change preceded the spike

    Recent deploys and config changes are the most common cause of sudden latency shifts, so correlating the spike with the change timeline is high-yield. Rewriting code first skips diagnosis entirely.

    Read the full bite: First steps on a p99 latency page

20. Question 20 of 30

    When designing an A/B test to measure how API latency impacts D7 retention, which setup best isolates causality?

    • aCompare historical users with naturally high latency against those with low latency from observational logs
    • bInject delays client-side on a subset of endpoints and use mean latency as the primary success metric
    • cRandomize each API request independently so the same user may experience both fast and slow responses
    • dBucket users by user ID, inject a consistent server-side delay per bucket, and track D7 retention alongside P99 latency
    Show the answer

    Answer: d · Bucket users by user ID, inject a consistent server-side delay per bucket, and track D7 retention alongside P99 latency

    User-level randomization with consistent server-side delays ensures the only systematic difference between groups is the injected latency, establishing causality. Comparing historical slow versus fast requests is confounded by variables like device type or network quality, so retention differences cannot be attributed solely to latency.

    Read the full bite: How would you design an experiment measuring API latency impact on retention?

21. Question 21 of 30

    Which statistical measure best reflects the typical user's experience when analyzing API response times?

    • aThe mean, as it provides an average of all recorded response times.
    • bThe mode, as it identifies the most common response time observed.
    • cThe median, because it is robust to extreme values caused by outliers.
    • dThe 99th percentile (p99), to ensure the vast majority of users have a good experience.
    Show the answer

    Answer: c · The median, because it is robust to extreme values caused by outliers.

    The median is preferred because API latency data often contains outliers (e.g., due to GC pauses) that can significantly skew the mean, misrepresenting the typical user experience. While the 99th percentile is crucial for understanding tail latency, it does not represent the 'typical' experience as effectively as the median.

    Read the full bite: Mean vs. Median for API Response Times?

22. Question 22 of 30

    Which approach best addresses the limitations of using only median latency for API monitoring?

    • aAdd p95, p99, and maximum to capture tail behavior and worst-case performance
    • bAverage p99 values across hosts to obtain a global tail-latency metric
    • cUse standard deviation around the median to quantify multi-modal variance
    • dSwitch to mean latency because it accounts for every request in the dataset
    Show the answer

    Answer: a · Add p95, p99, and maximum to capture tail behavior and worst-case performance

    The card advocates tracking p95, p99, and max because median alone discards the upper half of the distribution and hides tail latency. Distractor A is wrong because the mean is easily skewed by outliers like GC pauses, so accounting for every request actually misrepresents typical performance.

    Read the full bite: Mean or median for API response times?

23. Question 23 of 30

    When improving API p99 from 500ms to 200ms in a distributed system, which validation strategy best ensures real user benefit without hidden side effects?

    • aDeploy fine-grained edge histograms, propagate trace context across hops, and watch error rates, throughput, and cost
    • bHorizontally scale the service until the p99 drops to 200ms without checking for hot keys or lock contention
    • cAggregate individual backend p99 metrics into a global average and declare success when that average reads 200ms
    • dAdd aggressive timeouts to fail the slowest 1% of requests so they are excluded from the p99 calculation
    Show the answer

    Answer: a · Deploy fine-grained edge histograms, propagate trace context across hops, and watch error rates, throughput, and cost

    The correct strategy uses histograms to accurately detect true tail shifts, trace context to pinpoint which backend hop inflates latency, and complementary metrics to guard against side effects. Option D is tempting because it artificially lowers the percentile, but it hides latency by converting slow requests into errors, which directly harms user experience.

    Read the full bite: Monitor p99 improvement from 500ms to 200ms and side effects

24. Question 24 of 30

    Why is a t-test inappropriate for comparing p99 latency between two groups, and what is a suitable alternative?

    • aThe Central Limit Theorem ensures normality for large samples, but p99 is an outlier metric; therefore, a t-test on the raw latency data is still valid.
    • bA t-test relies on the sampling distribution of the mean being normal, which doesn't apply to p99; instead, bootstrap the difference in p99s to construct a confidence interval.
    • cP99 values are not normally distributed, so they should be log-transformed before applying a standard t-test.
    • dLatency data is inherently non-normal, making the t-test invalid; a non-parametric test like the Mann-Whitney U test should be used instead.
    Show the answer

    Answer: b · A t-test relies on the sampling distribution of the mean being normal, which doesn't apply to p99; instead, bootstrap the difference in p99s to construct a confidence interval.

    The t-test assumes the sampling distribution of the statistic (e.g., the mean) is approximately normal, which is not true for p99. Bootstrapping allows constructing a confidence interval for the difference in p99s by repeatedly resampling the original data. Log-transforming a single p99 value (option C) is incorrect; transformations apply to raw data.

    Read the full bite: Why not t-test p99 latency? Describe a valid alternative.

25. Question 25 of 30

    Why is a two-sample t-test fundamentally unsuitable for comparing p99 latency between treatment and control, and what is a valid alternative?

    • aA t-test assumes normally distributed individual observations, which latency violates; instead, use a Mann-Whitney U test on the raw latency values.
    • bThe sampling distribution of p99 is approximately normal, so a z-test for quantiles is valid without estimating the local density at the 99th percentile.
    • cThe Central Limit Theorem does not apply to order statistics, but you can preserve validity by applying the t-test only to the top one percent of observations.
    • dA t-test targets the sample mean, whereas p99 is an order statistic whose variance depends on local tail density; use bootstrap confidence intervals or permutation tests.
    Show the answer

    Answer: d · A t-test targets the sample mean, whereas p99 is an order statistic whose variance depends on local tail density; use bootstrap confidence intervals or permutation tests.

    A t-test is derived for the sample mean under the CLT, while p99 is an order statistic whose standard error is driven by the local density at the tail, making resampling methods like bootstrap confidence intervals or permutation tests the valid approach. Option C is tempting because it correctly notes the CLT does not apply to order statistics, but isolating the top one percent destroys the overall sample structure and does not fix the fundamental mismatch between a mean test and a quantile estimate.

    Read the full bite: Why can't you t-test p99 latency, and what's a valid alternative?

26. Question 26 of 30

    Why is a standard t-test an invalid method for determining if p99 latency has significantly changed in an A/B test?

    • aA t-test is designed to compare means, and the p99 is a measure of the tail, not the central tendency.
    • bLatency data is rarely normally distributed, and a t-test requires the underlying data to be normal.
    • cA single p99 value is calculated for each group, but a t-test requires a distribution of values from each group to run.
    • dThe sampling distribution of a sample percentile is not guaranteed to be normal, violating a core assumption of the t-test.
    Show the answer

    Answer: d · The sampling distribution of a sample percentile is not guaranteed to be normal, violating a core assumption of the t-test.

    The t-test relies on the Central Limit Theorem, which guarantees a normal sampling distribution for the mean, but this does not apply to percentiles. Option B is a common misconception; the t-test for comparing means is robust to non-normal data in large samples.

    Read the full bite: Why can't we t-test p99 latency in an A/B test?

27. Question 27 of 30

    When p95 TTFT exceeds SLO while per-token latency remains healthy, which remediation should you prioritize before changing model architecture?

    • aImmediately deploy a smaller or quantized model variant
    • bIncrease the dynamic batching token limit to maximize throughput
    • cAdd GPU replicas to distribute request load
    • dInspect queuing depth and right-size batching limits based on tracing data
    Show the answer

    Answer: d · Inspect queuing depth and right-size batching limits based on tracing data

    High TTFT with healthy per-token latency signals queuing or batching bottlenecks rather than raw execution speed, so structured debugging targets queue depth and batch limits first. Adding replicas assumes compute saturation without profiling, and increasing batch limits can actually worsen TTFT by delaying first-token delivery.

    Read the full bite: How would you systematically debug an inference API latency breach?

28. Question 28 of 30

    What directly causes the extra latency of a serverless cold start?

    • aThe database rejecting the connection until a warm instance retries
    • bProvisioning a new execution environment and initializing the runtime, code, and dependencies
    • cNetwork round-trip time between the client and the function endpoint
    • dThe garbage collector pausing during steady-state request handling
    Show the answer

    Answer: b · Provisioning a new execution environment and initializing the runtime, code, and dependencies

    A cold start is the time to allocate a fresh environment and run runtime and code initialization when no warm instance exists. It is unrelated to ordinary network latency or steady-state GC pauses.

    Read the full bite: Cold starts in serverless environments

29. Question 29 of 30

    A team must reprocess 30 days of sensor data through an updated model. Which serving pattern and infrastructure choice best fits this workload?

    • aDeploy an autoscaling API endpoint with low-latency feature store lookups to process the backlog synchronously
    • bUse a scheduled workflow on spot instances that partitions data and optimizes for throughput over per-request latency
    • cStream each historical record through a real-time prediction service to maintain strict P99 latency guarantees
    • dRun the job on always-on GPU instances with warm pools to avoid cold starts during the historical reprocessing
    Show the answer

    Answer: b · Use a scheduled workflow on spot instances that partitions data and optimizes for throughput over per-request latency

    This workload is classic batch inference: large historical data processed asynchronously without a client waiting, best served by scheduled workflows on spot instances optimizing throughput. Option A incorrectly applies online feature-store patterns to a backlog, while C and D misuse real-time serving infrastructure and SLAs for an offline job.

    Read the full bite: Describe the difference between online and batch inference.

30. Question 30 of 30

    While diagnosing high latency in an online inference service, you see p99 latency rising sharply while GPU utilization remains flat. What is the most appropriate next step?

    • aImmediately apply quantization to reduce model compute requirements
    • bInspect queue depth, batch size configuration, and autoscaling behavior
    • cProfile the model with a benchmark tool like trtexec to find theoretical minimum inference time
    • dCheck for cache misses and serialization overhead in the application layer
    Show the answer

    Answer: b · Inspect queue depth, batch size configuration, and autoscaling behavior

    Flat GPU utilization alongside rising tail latency strongly signals a queuing bottleneck rather than a compute-bound model, so you should inspect batching and autoscaling first. Jumping straight to quantization is a common mistake because it assumes the model is saturated when the GPU is not.

    Read the full bite: How would you systematically diagnose high latency in an online inference service?