Inference
42 bites tagged Inference — interview questions with model answers, and 60-second explainers.
Model Compilation: Bridging Models and Hardware
An ML compiler translates a model's abstract math into optimized instructions for specific hardware. This lets you run the same model efficiently on cloud GPUs, mobile CPUs, or edge devices.
Inference Batching: Grouping Requests for Throughput
Think of inference batching as a carpool for your ML model. Instead of sending each request in its own car, you wait a few microseconds to fill a bus, dramatically improving GPU efficiency.
NVIDIA Triton: A Universal AI Model Server
Triton Inference Server is like a universal remote for AI models, providing a standard API to serve models from any framework. Use it to deploy diverse models (PyTorch, ONNX) without custom serving stacks.
Streaming Inference: Real-Time Model Predictions
Streaming inference makes predictions on data in-flight, not from a database. It's for real-time recommendations or fraud detection where millisecond decisions are critical. The footgun is assuming a single server can handle the load; you must build for scale.
Serverless Inference: Run ML Models Without Managing Servers
Serverless inference treats ML prediction like a function call, abstracting away servers. You pay for compute time per prediction, not for idle infrastructure.
Batch Inference: High Throughput, Not High Speed
Think of batch inference as processing a day's mail at once, not as each letter arrives. It trades immediate answers for cost-effective, high-volume predictions, like generating daily product recommendations. The footgun is using it for real-time needs.
Online Inference: Predictions on Demand
Online inference is a vending machine for predictions: you make one request and get one result back immediately. It powers real-time features like fraud detection or content moderation.
Quantization-Aware Training (QAT): Forcing Models to Learn While Quantized
QAT forces a model to "learn its own compression" by simulating quantization during training. This lets you shrink LLMs to aggressive low-bit formats (like 4-bit) where simpler post-training methods fail.
Dynamic Batching: Balancing LLM Throughput and Latency
Dynamic batching groups LLM requests like a bus that leaves on a schedule or when full, whichever comes first. This improves throughput in inference servers by avoiding long waits. The footgun: all requests in a batch are still held hostage by the slowest one.
vLLM: Faster LLM Inference with PagedAttention
vLLM is a serving engine that speeds up LLM inference by treating GPU memory like virtual memory. It's used to serve models with higher throughput by batching requests without wasting memory on padding.
Speculative Decoding: A Small LLM Speeds Up a Big One
Speculative decoding uses a small, fast 'draft' LLM to guess the next few words, which a larger 'target' LLM verifies in one batch. This cuts inference latency 2-3x in production systems.
ONNX Runtime: Run Any AI Model, Anywhere
ONNX Runtime is a universal engine for AI models, letting you run them efficiently on any hardware, from cloud GPUs to a user's browser. It's used to deploy models for fast inference on servers or mobile devices.
KV Cache: Don't Recompute, Just Remember
KV Cache speeds up LLM text generation by storing intermediate calculations (Key/Value vectors) instead of recomputing them for every new token. It's a standard optimization in inference engines.
TensorRT: From Trained Model to Production Speed
TensorRT is a compiler that turns a trained model into a specialized, high-speed engine for a specific NVIDIA GPU. It's used to deploy models in production where low latency is critical.
Edge AI Accelerators: Inference Without the Cloud
An Edge AI accelerator is a specialized, low-power chip that runs AI models directly on a device, skipping the cloud. It's used for real-time tasks like object detection where latency and privacy are critical.
Model Quantization: Trading Precision for Performance
Model quantization trades numerical precision for a smaller memory footprint. It reduces model weights from high-precision types like fp32 to lower ones like int8 or int4, making large models fit on consumer hardware.
What is the KV cache and why does it matter for serving LLMs?
This question tests your understanding of performance bottlenecks in autoregressive LLM inference. A great answer first explains that the attention mechanism computes Key (K) and Value (V) tensors for all input tokens. Then, it highlights the redundancy of recomputing these for past tokens at each new generation step. The KV cache solves this by storing these tensors, drastically reducing latency. A red flag is vaguely calling it a 'cache' without connecting it to K/V tensors.
Speculative Decoding: Faster LLM Inference, Same Results
Speculative decoding accelerates LLM inference by using a small, fast "draft" model to predict a sequence of tokens. The large, accurate model then validates this entire sequence in a single parallel pass, instead of generating one token at a time. This is used to get 2-3x speedups on production models without retraining. The common misconception is that it's a lossy approximation; in reality, it produces bit-for-bit identical output to the original model.
Get Inference bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.