Top 30 Inference Interview Questions and Answers
30 multiple-choice questions on Inference, drawn from 30 bites out of the 42 tagged Inference on Tezvyn. Answer them here or read straight down. Every question carries the correct option, why it is correct, and a link to the bite it came from.
30 questions. Pick an answer, or open “Show the answer” to read it.
Answers are graded in your browser. Nothing is saved, and no XP or streak is earned here. The app keeps score.
Question 1 of 30
What justifies using a z-test for a population mean when the underlying data are heavily skewed?
Show the answer
Answer: a · The sampling distribution of the sample mean becomes approximately normal for large n
The CLT states that the sampling distribution of the sample mean approaches normality as n grows, which justifies using z-tests even when the population is skewed. Option B describes the Law of Large Numbers, a common look-alike that explains convergence to a single value rather than the bell-curve shape required for inference.
Read the full bite: Explain the Central Limit Theorem and its importance for hypothesis testing
Question 2 of 30
An engineer fits a distribution to server latency using MLE and reports tight confidence intervals. If the chosen distribution family does not match the true data-generating process, what best describes the result?
Show the answer
Answer: b · The estimates maximize likelihood within the wrong family and can be precisely misleading
C is correct because MLE finds the parameters that make the observed data most probable within the assumed model, so a wrong family yields a precise but misleading fit. D is tempting but wrong because misspecification does not automatically inflate uncertainty; the method can be confidently wrong.
Read the full bite: MLE: Find the Parameters That Make Data Likely
Question 3 of 30
In which situation should you avoid few-shot prompting?
Show the answer
Answer: d · A clearly described one-sentence task is running near the model's token limit
The card states that few-shot prompting should be avoided when a zero-shot instruction is sufficient and when the context window is nearly full, because examples then add cost and bias without benefit. Distractor D describes a valid use case, as adapting niche style without retraining is exactly where few-shot prompting excels.
Question 4 of 30
Which statement best captures a fundamental architectural difference between online and batch inference?
Show the answer
Answer: c · Online uses always-on endpoints optimizing for latency, while batch uses elastic compute optimizing for throughput.
Online inference serves live requests via always-on endpoints optimized for low latency and availability, while batch inference asynchronously processes large datasets on elastic compute that scales to zero, prioritizing throughput and total cost. The most tempting distractor incorrectly treats batch as merely slow online inference with queued requests, ignoring that batch is a fundamentally asynchronous paradigm driven by data volume rather than per-request response times.
Read the full bite: Describe the difference between online and batch inference.
Question 5 of 30
A team must reprocess 30 days of sensor data through an updated model. Which serving pattern and infrastructure choice best fits this workload?
Show the answer
Answer: b · Use a scheduled workflow on spot instances that partitions data and optimizes for throughput over per-request latency
This workload is classic batch inference: large historical data processed asynchronously without a client waiting, best served by scheduled workflows on spot instances optimizing throughput. Option A incorrectly applies online feature-store patterns to a backlog, while C and D misuse real-time serving infrastructure and SLAs for an offline job.
Read the full bite: Describe the difference between online and batch inference.
Question 6 of 30
When building a containerized REST API to serve a trained model, which approach aligns with production MLOps best practices?
Show the answer
Answer: b · Load the model once at startup, expose /health and /predict routes, and serve with Uvicorn inside a slim container.
Loading the artifact once at startup and serving it with a production-grade server via dedicated health and predict endpoints follows the minimal viable serving lifecycle. Option A is tempting because it sounds cautious, but reloading the model per request causes catastrophic latency.
Read the full bite: Deploy a trained model as a containerized REST API
Question 7 of 30
Which practice is essential when validating a quantized model before promoting it to production traffic?
Show the answer
Answer: a · Run downstream task benchmarks and shadow A/B against the full-precision model.
Downstream benchmarks and shadow A/B are required to catch inference-time regressions that aggregate metrics miss. File-size reduction alone is tempting because it confirms compression occurred, yet it says nothing about model quality or task performance.
Read the full bite: Explain model quantization, its benefits, drawbacks, and validation approach
Question 8 of 30
Which scenario best exemplifies the core application of online inference?
Show the answer
Answer: d · A model that processes a user's search query and instantly returns personalized results.
Online inference is characterized by providing immediate, single-prediction responses to individual requests, as seen when a user's search query instantly yields results. Option B describes streaming analytics, which is real-time but involves continuous processing of data streams rather than discrete, on-demand predictions for single events.
Question 9 of 30
For which task would batch inference be the most suitable approach?
Show the answer
Answer: b · Generating daily personalized product recommendations for all users.
Batch inference is designed for processing large volumes of data on a schedule when immediate results are not required, as exemplified by generating daily recommendations. Real-time fraud detection, interactive chatbots, and dynamic ad bidding all require immediate, low-latency responses, making them unsuitable for batch inference.
Read the full bite: Batch Inference: High Throughput, Not High Speed
Question 10 of 30
Which scenario is most appropriate for deploying an ML model using serverless inference?
Show the answer
Answer: b · An internal application that summarizes user-uploaded documents only a few times a day.
Serverless inference is ideal for workloads with intermittent, infrequent, or unpredictable traffic, as it allows resources to scale down to zero when not in use, saving costs. The card specifically mentions an "internal tool for summarizing documents on demand" as a suitable use case. Options A, B, and D describe scenarios with high, sustained traffic or strict low-latency requirements, for which serverless is explicitly not recommended due to cold starts and cost-inefficiency compared to provisioned endpoints.
Read the full bite: Serverless Inference: Run ML Models Without Managing Servers
Question 11 of 30
For which task would streaming inference typically be considered an inefficient or unsuitable approach?
Show the answer
Answer: c · Running complex analytics on a full day's worth of server logs for daily insights.
Streaming inference is designed for immediate, low-latency predictions on data in motion, as exemplified by options A, B, and D. Option C describes a task better suited for batch processing, as it involves analyzing a large, complete dataset over a longer period, which can tolerate latency and where the operational complexity of streaming is unnecessary.
Read the full bite: Streaming Inference: Real-Time Model Predictions
Question 12 of 30
What core challenge in AI model deployment does NVIDIA Triton primarily address?
Show the answer
Answer: b · Managing the complexity and inefficiency of serving models from diverse AI frameworks.
Triton was designed to solve the inefficiency and error-proneness of building and maintaining separate serving applications for models from different frameworks. While it provides a standardized API, that is a solution Triton offers to address this underlying complexity, not the primary problem itself.
Read the full bite: NVIDIA Triton: A Universal AI Model Server
Question 13 of 30
What is the primary trade-off when quantizing an LLM to INT4 for deployment?
Show the answer
Answer: c · Smaller memory and faster inference at the cost of potential accuracy degradation
Quantization shrinks memory and speeds inference but lower precision risks accuracy loss, sharper at INT4. It targets inference not training, and is not lossless, so the other options are wrong.
Question 14 of 30
How does the KV cache reduce the cost of autoregressive generation?
Show the answer
Answer: d · It stores past tokens' key and value vectors so each step computes only the new token rather than recomputing the whole prefix
Past keys and values are fixed, so caching them avoids recomputing the prefix each step, making per-token cost linear. It caches K and V tensors, not outputs or weights, and applies to inference, not training.
Read the full bite: How the KV cache speeds transformer generation
Question 15 of 30
What is the primary trade-off when implementing dynamic inference batching for an ML model?
Show the answer
Answer: b · It enhances overall hardware throughput by introducing a slight increase in individual request latency.
Dynamic inference batching groups multiple requests to improve GPU utilization and throughput, but this process inherently introduces a small, configurable delay for individual requests. Option D is incorrect because batching actually increases individual request latency, making it unsuitable for ultra-low-latency applications.
Read the full bite: Inference Batching: Grouping Requests for Throughput
Question 16 of 30
How is classifier-free guidance enabled during training?
Show the answer
Answer: c · By randomly dropping the conditioning signal so one model learns conditional and unconditional denoising
Randomly nulling the condition lets a single network learn both conditional and unconditional denoising, which are combined at inference. Training a separate noisy-image classifier describes classifier guidance, the method it replaces.
Read the full bite: Classifier-free guidance in diffusion models
Question 17 of 30
What does PagedAttention primarily improve in LLM serving?
Show the answer
Answer: d · KV-cache memory utilization by allocating non-contiguous fixed-size blocks on demand
PagedAttention is a memory-management technique that eliminates fragmentation from contiguous max-length allocation, raising throughput. It does not change the attention math, precision, or sampling quality.
Read the full bite: What memory problem PagedAttention solves
Question 18 of 30
Why is tensor parallelism typically kept within a single node while pipeline parallelism spans nodes?
Show the answer
Answer: b · Tensor parallelism all-reduces every layer needing fast interconnect; pipeline parallelism only passes activations at stage boundaries
Tensor parallelism's frequent high-volume all-reduces demand NVLink-class links, so it stays intra-node; pipeline parallelism's lighter boundary communication tolerates slower cross-node links. Neither replicates the full model, and they are commonly combined.
Read the full bite: Tensor versus pipeline parallelism for large models
Question 19 of 30
What is the primary advantage of employing model compilation in machine learning workflows?
Show the answer
Answer: d · It enables a single model definition to be efficiently deployed across various hardware platforms.
Model compilation's core purpose is to translate a high-level model into an optimized, hardware-specific artifact, solving the 'write once, run anywhere' problem for efficient inference across diverse devices. While compilation involves optimization, it is explicitly for inference performance, not training acceleration or facilitating early-stage architectural changes.
Read the full bite: Model Compilation: Bridging Models and Hardware
Question 20 of 30
For which application would optimizing inference throughput be the primary concern?
Show the answer
Answer: d · A system generating personalized recommendations for millions of users
Inference throughput measures the total number of predictions a system can make per second, which is critical for high-volume tasks like recommendation engines. The other options describe scenarios where low latency for individual requests is paramount, making throughput a secondary concern.
Read the full bite: Inference Throughput: How Many Predictions Per Second?
Question 21 of 30
In a top-2 Mixture of Experts layer, what primarily explains the lower per-token compute compared to a dense model of the same total parameter count?
Show the answer
Answer: d · Only the few experts selected by the router compute for each token
Sparse activation means just the top-k experts fire per token, so FLOPs scale with active parameters, not total. The last option is wrong because non-selected experts do not run at all.
Read the full bite: Mixture of Experts architecture and routing
Question 22 of 30
What is the primary mechanism by which KV Cache accelerates large language model (LLM) text generation?
Show the answer
Answer: d · It stores the Key and Value vectors of previously generated tokens, preventing their recomputation in subsequent steps.
KV Cache speeds up generation by storing the Key and Value vectors of past tokens, so the model doesn't have to recompute them for every new token. Option B is incorrect because KV Cache focuses on storing K and V vectors of *past* tokens, not pre-computing Q vectors for all tokens.
Read the full bite: KV Cache: Don't Recompute, Just Remember
Question 23 of 30
What is a key trade-off of using int8 quantization to speed up a detection model?
Show the answer
Answer: b · It can cause a small accuracy drop in exchange for faster, smaller inference
Quantization shrinks the model and speeds inference with usually minor accuracy loss, reducible via quantization-aware training. It does not boost accuracy, need full retraining, or exclude GPUs and accelerators.
Read the full bite: How do you speed up a slow detection model?
Question 24 of 30
What is the core function of ONNX Runtime in the AI model lifecycle?
Show the answer
Answer: a · To serve as a universal, high-performance engine for deploying trained AI models across diverse hardware platforms.
ONNX Runtime's primary role is to act as a high-performance inference engine, allowing trained AI models (in ONNX format) to be deployed and run efficiently on various hardware. While conversion to ONNX format is a prerequisite, ONNX Runtime itself is not a conversion tool between training frameworks, nor is it for training or experimentation.
Read the full bite: ONNX Runtime: Run Any AI Model, Anywhere
Question 25 of 30
Why can increasing batch size in an LLM inference server hurt a single user's experience even as it raises throughput?
Show the answer
Answer: a · Waiting to fill a larger batch adds queueing delay that increases time to first token
Throughput and latency trade off: forming a bigger batch means waiting for more requests, delaying each one's first token. Batch size does not change accuracy, and larger batches use more, not less, memory.
Read the full bite: Dynamic batching and the throughput-latency trade-off
Question 26 of 30
Which statement accurately describes how speculative decoding guarantees its output is identical to the target model's standalone generation?
Show the answer
Answer: c · The target model validates the draft's proposed tokens in a single pass, accepting only those it would have generated itself.
The core guarantee of speculative decoding comes from the target model's verification step, where it checks the draft's proposed tokens against what it would have generated itself and only accepts matching ones. The system does not discard the entire sequence upon an error; it discards only the incorrect suffix and continues generation from the point of divergence.
Read the full bite: Speculative Decoding: A Small LLM Speeds Up a Big One
Question 27 of 30
Why do a few large-magnitude activation outliers degrade INT8 quantization accuracy so much?
Show the answer
Answer: b · They force the quantization scale wider, squeezing normal values into too few integer levels
Outliers stretch the per-tensor range, so most values map to a narrow band of the 256 INT8 levels and lose precision. Keeping outlier channels in FP16 (mixed precision) restores accuracy. Weights are not corrupted by this.
Read the full bite: Handling outlier activations in INT8 quantization
Question 28 of 30
For a large deep-learning model, what condition most makes a GPU cost-effective for serving compared to a CPU?
Show the answer
Answer: d · High request volume that can be batched to keep the GPU utilized
GPUs pay off when batched high-volume traffic keeps their parallel units busy. Single tiny requests leave a GPU idle, strict cost minimization favors CPUs, and small classical models do not need GPU parallelism.
Read the full bite: CPU versus GPU serving: cost, latency, throughput
Question 29 of 30
Which configuration is essential for a real-time inference endpoint to handle variable production traffic cost-effectively?
Show the answer
Answer: d · Autoscaling that adjusts instance count to request volume
Autoscaling matches capacity to demand, avoiding both overload and idle waste. A fixed peak-sized instance wastes money off-peak, and disabling health checks or skipping canary rollout undermines reliability.
Read the full bite: Deploying a real-time inference endpoint
Question 30 of 30
vLLM's PagedAttention mechanism is analogous to virtual memory in an OS because it primarily addresses which challenge in LLM inference?
Show the answer
Answer: c · Efficiently managing the Key-Value (KV) cache by allowing non-contiguous memory allocation.
PagedAttention's core innovation is to manage the Key-Value (KV) cache by dividing it into smaller, non-contiguous blocks, similar to how virtual memory manages RAM. This enables efficient memory utilization for concurrent requests. Option D describes a consequence of improved efficiency, not the direct mechanism of PagedAttention itself.
Read the full bite: vLLM: Faster LLM Inference with PagedAttention
Could you explain these out loud?
That is what an interview actually tests. Tezvyn gives you questions like these with what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.