Optimization
48 bites tagged Optimization — interview questions with model answers, and 60-second explainers.
Model Compilation: Bridging Models and Hardware
An ML compiler translates a model's abstract math into optimized instructions for specific hardware. This lets you run the same model efficiently on cloud GPUs, mobile CPUs, or edge devices.
Multi-stage Docker Builds: Lean Images, Fast Deploys
Build your app in one container stage and run it in another, separate one. This keeps your final Docker image lean by shipping only the compiled artifact, not the entire build environment, compilers, and source code.
Dynamic Batching: Balancing LLM Throughput and Latency
Dynamic batching groups LLM requests like a bus that leaves on a schedule or when full, whichever comes first. This improves throughput in inference servers by avoiding long waits. The footgun: all requests in a batch are still held hostage by the slowest one.
vLLM: Faster LLM Inference with PagedAttention
vLLM is a serving engine that speeds up LLM inference by treating GPU memory like virtual memory. It's used to serve models with higher throughput by batching requests without wasting memory on padding.
Speculative Decoding: A Small LLM Speeds Up a Big One
Speculative decoding uses a small, fast 'draft' LLM to guess the next few words, which a larger 'target' LLM verifies in one batch. This cuts inference latency 2-3x in production systems.
FlashAttention: Faster, Memory-Efficient Exact Attention
FlashAttention is an IO-aware algorithm that computes exact attention faster and with less memory. It avoids slow GPU memory transfers, making it a key optimization for training and serving large models on modern GPUs.
ONNX Runtime: Run Any AI Model, Anywhere
ONNX Runtime is a universal engine for AI models, letting you run them efficiently on any hardware, from cloud GPUs to a user's browser. It's used to deploy models for fast inference on servers or mobile devices.
KV Cache: Don't Recompute, Just Remember
KV Cache speeds up LLM text generation by storing intermediate calculations (Key/Value vectors) instead of recomputing them for every new token. It's a standard optimization in inference engines.
Learning Rate Scheduling: A Gearbox for Model Training
Think of a learning rate schedule as a training 'gearbox,' starting fast and slowing for precision. It's used when fine-tuning large models to adapt them without breaking them.
Mixed-Precision Training: Faster Training with Less Memory
Mixed-precision training is like using rough estimates (FP16) for most math and a calculator (FP32) for critical steps. This speeds up deep learning on GPUs by cutting memory use, but naively switching can cause training to fail as small gradients vanish.
Loss Function: Quantifying 'How Wrong' a Model Is
A loss function is a score that tells a machine learning model how wrong its predictions are. The lower the score, the better. It's the engine of training, guiding the model to adjust its parameters to get closer to the correct answers.
Docker Multi-stage Builds: Slimmer, Faster Images
Treat your Dockerfile like a pipeline: build your app in one stage with all its tools, then copy only the final artifact to a clean production stage. This keeps images small by excluding build-time dependencies.
Predicate Pushdown: Filter Data at the Source
Predicate pushdown tells the database to filter data at the source, not after fetching it. This speeds up queries in data warehouses and lakehouses by reducing network traffic. The main footgun: not all data sources can execute all types of filters.
The Jacobian Matrix: A Derivative for Multiple Dimensions
The Jacobian matrix is the multi-dimensional version of a derivative. It's a grid of partial derivatives showing how a small change in each input locally affects each output of a vector function. Don't confuse the matrix with its determinant.
Gradient Descent: Finding the Bottom of the Hill
Think of finding the lowest point on a foggy hill by taking steps in the steepest downward direction. It's how machine learning models learn, by iteratively minimizing a cost function. The footgun is the step size: too large overshoots, too small is too slow.
Multi-Armed Bandits: Balancing Exploration and Exploitation
A multi-armed bandit algorithm balances exploiting a known winner with exploring new options to find a better one. It's used to dynamically test website headlines or ad copy, automatically shifting traffic to the best performer to maximize conversions.
Multivariate Testing (MVT): Finding the Best Combination
Multivariate testing (MVT) finds the best combination of elements, not just the best single version. It tests multiple headlines, images, and buttons at once to see how they interact. The main footgun is needing massive traffic for statistically valid results.
Dynamic Creative Optimization (DCO)
Dynamic Creative Optimization (DCO) is a technology for programmatic advertising. It allows advertisers to automatically optimize the performance of their creative content using real-time systems, rather than serving a single static ad.
TensorRT: From Trained Model to Production Speed
TensorRT is a compiler that turns a trained model into a specialized, high-speed engine for a specific NVIDIA GPU. It's used to deploy models in production where low latency is critical.
Batch Normalization: Stabilizing Neural Network Training
Batch Normalization regulates data flow in a neural network by re-centering and re-scaling inputs to each layer. This stabilizes deep network training, allowing higher learning rates.
Backpropagation: How Neural Networks Learn from Mistakes
Backpropagation is how a network learns from its mistakes. It works backward from the output error, calculating how much each weight contributed and adjusting it. This is the core training loop for most deep learning models.
Cloud Rightsizing: Stop Overpaying for VMs
Rightsizing stops you from overpaying for idle cloud capacity. It involves analyzing CPU and memory usage to shrink over-provisioned VMs. Always collaborate with application owners before making changes.
Multi-Armed Bandit: The Explore vs. Exploit Trade-off
A multi-armed bandit algorithm balances exploring new options with exploiting the current winner, like a gambler trying slot machines to find the best payout.
Speculative Decoding: Faster LLM Inference, Same Results
Speculative decoding accelerates LLM inference by using a small, fast "draft" model to predict a sequence of tokens. The large, accurate model then validates this entire sequence in a single parallel pass, instead of generating one token at a time. This is used to get 2-3x speedups on production models without retraining. The common misconception is that it's a lossy approximation; in reality, it produces bit-for-bit identical output to the original model.
Get Optimization bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.