Latency
47 bites tagged Latency — interview questions with model answers, and 60-second explainers.
Why tail latency (p99) matters
Averages hide the worst experiences, and fan-out amplifies tails so most requests touch a slow path; causes include GC pauses, queueing, contention, and noisy neighbors. Tail-latency reasoning.
Diagnose a degraded canary release
Check statistical significance versus baseline, confirm apples-to-apples comparison, isolate the cause via traces and logs, then weigh the regression against SLO budget. Canary decision-making.
First steps on a p99 latency page
Confirm scope and blast radius, check recent changes and deploys, then look at dependencies and resource saturation. a structured triage instinct, not random poking. diving into code or restarting things before scoping impact.
Writing SLIs in PromQL
Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le. turning SLOs into correct PromQL.
Prometheus histogram versus summary
Histograms expose bucket counts and let you compute aggregatable quantiles at query time; summaries compute fixed quantiles per instance that cannot be combined. where quantiles are computed and whether they aggregate.
Histograms versus summaries for latency
Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged. understanding where quantiles are computed and whether they aggregate.
Essential tags for a request latency metric
Add method, route template, status code, and maybe service or region; they enable slicing while staying low-cardinality. Choosing useful, bounded metric labels. Adding user IDs or raw URLs, exploding cardinality.
Diagnosing a healthy p50 but breaching p99
One percent of requests are slow, hurting power users and fan-out calls; investigate GC, locks, contention, cold caches, retries. Understanding tail latency. Dismissing it because the average looks fine.
Proposing availability and latency SLIs for an auth API
Availability as the ratio of successful valid requests; latency as the fraction served under a threshold; measure at the edge from the user's view. Designing measurable, user-centric SLIs.
Why use latency percentiles over the average?
Averages hide the tail and are skewed by outliers, so most users can suffer while the mean looks fine; use percentiles; prioritize p99 when tail users are high-value or fan-out… understanding of latency distributions and tail behavior.
What are the Four Golden Signals?
Name latency, traffic, errors, and saturation; explain each briefly; describe measuring latency as a distribution separating success from failure. knowledge of the core user-facing monitoring signals.
Dynamic batching and the throughput-latency trade-off
The server groups concurrent requests into one batch to use the GPU fully, but larger batches and waiting to fill them raise per-request latency and time to first token. how batching balances GPU efficiency and latency.
Three techniques to cut LLM inference latency
Quantization shrinks weights with small accuracy risk, KV-cache plus continuous batching boost throughput, speculative decoding drafts tokens for lossless speedup. knowledge of software-level inference optimization.
Designing a production LLM summarization eval
A representative gold set, quality via human or LLM-as-judge plus faithfulness checks, and operational metrics like p95 latency and cost per request. building task-specific eval beyond leaderboards.
Pinning exclusive CPU cores to a pod
Set kubelet CPU Manager policy to static, make the pod Guaranteed QoS with integer CPU limits equal to requests, so it gets exclusive dedicated cores. Achieving CPU pinning.
Diagnosing database latency layer by layer
Split total time into pool-wait, query execution, and ORM-generated query patterns; use metrics at each layer. structured latency diagnosis.
Tuning a database connection pool
Max size, min idle, connection and max-lifetime timeouts; size from cores and latency, not guesswork. connection-pool sizing intuition.
Minimizing model prediction latency end to end
Shrink the model via quantization, pruning, distillation, compilation; cut infra latency via caching, edge, faster runtimes. latency optimization across layers. only buying bigger hardware or ignoring accuracy tradeoffs.
Batch prediction versus online real-time prediction
Batch is scheduled bulk scoring, online is low-latency per-request scoring; contrast latency, freshness, cost; give a use case each. serving pattern selection. confusing batch with retraining or claiming online is always best.
Challenges deploying a model on edge hardware
Limited memory and compute cause latency, thermal and power limits, accuracy loss from compression, operator support gaps. edge deployment constraints. only mentioning accuracy while ignoring latency, memory, and power.
Serverless cold starts and how to mitigate them
A cold start is the latency to provision and initialize a fresh environment; mitigate with provisioned concurrency, smaller packages, and lighter runtimes. the serverless execution model. calling every slow call cold.
Cold starts in serverless environments
A cold start is the delay to provision a fresh instance and initialize the runtime; mitigate with provisioned concurrency and by shrinking init work. serverless latency internals. blaming network or steady-state latency.
Accelerating uncacheable dynamic traffic globally
Terminate TLS at a nearby edge and ride the provider backbone via Global Accelerator or CDN dynamic acceleration; add edge compute; ultimately deploy multi-region. accelerating non-cacheable traffic.
Diagnosing latency with distributed tracing
Follow the trace ID across spans, compare per-span durations to find the slow hop, distinguish service time from network and queueing. using traces to localize a bottleneck.
Get Latency bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.