Inference
42 bites tagged Inference — interview questions with model answers, and 60-second explainers.
Flask/Gunicorn vs Triton/TorchServe for serving
Flask is simple and flexible but lacks dynamic batching, GPU scheduling, and multi-model management; Triton/TorchServe add those plus metrics and versioning. choosing serving infrastructure.
Handling outlier activations in INT8 quantization
Profile activation magnitudes to find a few large-magnitude outlier channels, then keep those in higher precision while quantizing the rest, a mixed-precision decomposition. why INT8 quantization fails and how to fix it.
Dynamic batching and the throughput-latency trade-off
The server groups concurrent requests into one batch to use the GPU fully, but larger batches and waiting to fill them raise per-request latency and time to first token. how batching balances GPU efficiency and latency.
Mixture of Experts architecture and routing
Many expert FFNs per layer, a router picks top-k experts per token, only those compute so active params are far fewer than total. grasp of sparse activation and the gating router.
Tensor versus pipeline parallelism for large models
Tensor parallelism splits individual layers across GPUs needing fast interconnect; pipeline parallelism splits layers into stages across GPUs. multi-GPU model sharding strategy.
What memory problem PagedAttention solves
Pre-allocating contiguous max-length cache per sequence wastes memory through internal and external fragmentation; PagedAttention stores KV in fixed non-contiguous blocks like OS paging. KV-cache memory management at serving scale.
How the KV cache speeds transformer generation
Cache stores past keys and values so each new token only computes its own K, Q, V instead of recomputing all prior tokens, cutting per-step cost from quadratic to linear. understanding attention during decoding.
Model quantization benefits and risks
Quantization stores weights and activations in lower precision like INT8 or INT4; benefits are smaller memory and faster, cheaper inference; risk is accuracy loss. deployment-efficiency fundamentals.
CPU versus GPU serving: cost, latency, throughput
GPUs win on throughput for batched parallel work but cost more; CPUs suit low-volume or small models. inference hardware tradeoffs. claiming GPU is always faster or ignoring batching and utilization.
How do you speed up a slow detection model?
Quantization, pruning, distillation, lighter backbones, and resolution or batching tweaks, each trading some accuracy or effort for speed. inference optimization techniques.
Classifier-free guidance in diffusion models
Train one model jointly on conditional and dropped-condition inputs; at inference extrapolate from unconditional toward conditional prediction via a guidance scale, sharpening prompt adherence… how guidance improves conditioning.
Inference performance bottlenecks on Lambda
Cold starts loading the model, memory and CPU limits, no GPU, and package size dominate; mitigate with provisioned concurrency, loading the model once outside the handler, smaller models, and right-sized… serverless ML serving limits.
Deploying a real-time inference endpoint
Package the model artifact and inference code in a container, choose instance type and autoscaling, configure the endpoint with health checks, and plan safe rollout like canary plus monitoring. model serving operations.
Explain dynamic batching in inference servers and its trade-off
Dynamic batching launches when a time window or max size is met, improving throughput over static batching, but short ones wait for the slowest. Inference scheduling and the latency-vs-throughput trade-off.
Explain model quantization, its benefits, drawbacks, and validation approach
Tests precision trade-offs in production. Answer: define lowering weights from fp32 to int8/int4; cite memory and latency gains versus accuracy loss; validate with downstream benchmarks and shadow A/B. Red flag: treating as lossless or skipping task metrics.
Deploy a trained model as a containerized REST API
This tests practical MLOps fluency. A strong answer covers loading the artifact, wrapping it in a web server, building a Dockerfile, and exposing health and predict endpoints. Red flag: conflating training with serving or omitting resource limits.
Describe the difference between online and batch inference.
Online uses autoscaling APIs for millisecond-to-second latency; batch uses scheduled compute for minute-to-hour latency. Your grasp of serving patterns and infra tradeoffs.
Describe the difference between online and batch inference.
Online serves single requests in ms on live endpoints; batch processes data offline with elastic compute. Latency vs throughput trade-offs in serving. Treating batch as slow online inference or ignoring SLA and cost.
Inference Health Checks: Traffic Gates, Not Heartbeats
An inference server's health check is a traffic gate, not a heartbeat. Kubernetes uses it to route requests only after the model is loaded. The footgun is probing the root path, which stays green even when the model has crashed or the GPU is wedged.
Right-Size Inference and Stop Paying for Idle GPUs
Instance right-sizing matches inference to the smallest hardware that serves it without choking. It matters when GPU endpoints idle at 10% utilization. The footgun is copying your training spec into production; inference rarely needs that memory or multi-GPU.
Few-Shot Prompting
Few-shot prompting embeds task examples directly in the prompt to guide output format without retraining. It excels at niche tasks and consistent formatting, but mismatched examples degrade performance more than no examples at all.
Explain the Central Limit Theorem and its importance for hypothesis testing
This tests whether you know why sample means from skewed populations tend toward normal as size grows, enabling tests. A strong answer covers mean convergence to normal and standard error. Red flag: claiming the CLT works for small samples or single points.
MLE: Find the Parameters That Make Data Likely
MLE tunes your model until observed data looks inevitable. Use it to fit distributions to logs, traffic, or errors. The footgun: it assumes your distribution family is correct; under a wrong model, it finds the best-fitting wrong answer with high confidence.
Inference Throughput: How Many Predictions Per Second?
Inference throughput measures how many predictions your system can make per second, not how fast a single one is. It's the system's total capacity, critical for high-volume tasks like recommendation engines. The footgun is confusing it with latency.
Get Inference bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.