Intermediate everything in MLOps & Infrastructure, page 5
Adversarial Validation: Detect Drift with a Classifier
Adversarial validation reframes drift detection as a classification problem: train a model to separate training rows from production rows. If it can tell them apart, your feature distributions have shifted.
Inference Health Checks: Traffic Gates, Not Heartbeats
An inference server's health check is a traffic gate, not a heartbeat. Kubernetes uses it to route requests only after the model is loaded. The footgun is probing the root path, which stays green even when the model has crashed or the GPU is wedged.
Right-Size Inference and Stop Paying for Idle GPUs
Instance right-sizing matches inference to the smallest hardware that serves it without choking. It matters when GPU endpoints idle at 10% utilization. The footgun is copying your training spec into production; inference rarely needs that memory or multi-GPU.
Why GPUs Dominate Neural Network Training
A GPU is a freight train, a CPU a race car: deep learning moves identical math across huge batches. GPUs win on transformers and CNNs. The footgun is using them for tiny models, where data transfer overhead eats the gains.
Docker Image vs. Container: Blueprint vs. Runtime
A Docker image is a read-only blueprint; a container is a live instance with a writable layer. You build an image once in CI and run many containers from it in production. The footgun is mutating a running container without updating the image recipe.
MLflow Models Standardize Deployment Packaging
MLflow Models wrap artifacts into a standard package so one pipeline serves sklearn or PyTorch without new deployment code. Teams ship experiments to REST endpoints without Dockerfiles per model. Missing dependency logging lets model load but fail to predict.
Experiment Run: The Immutable Training Receipt
An experiment run is an auto-generated log for one training job: it captures hyperparameters, metrics, code, and artifacts. Teams use runs to debug regressions and audit settings. The footgun is logging many metrics without versioning data so comparison fails.
Log Transformation: Compress the Long Tail
Log transformation compresses the long tail of skewed data so outliers cannot dominate loss. Use it for features like income or latency that span orders of magnitude. The footgun is blindly applying it to zeros or negatives, which destroys data.
Model Registry: Source of Truth for Deployed Models
A model registry is the source of truth for which trained model runs where, turning anonymous artifact files into versioned, staged assets. It matters when you deploy multiple models or need instant rollbacks.
Model Server: The MLOps Deployment Bridge
A model server bridges ML training and production, operationalizing models within your release cycle. Use it when models must become first-class CI/CD citizens. The footgun is treating deployment as a one-time handoff rather than repeatable infrastructure.
Shadow Deployment: Test Models on Real Traffic
Shadow deployment runs a new model on real traffic without serving its predictions, letting you catch data drift before users are affected. It is the safest production validation method, but teams often forget to monitor its latency and resource costs.
MLOps: When to Build vs. Buy Your Infrastructure
Deciding to build or buy MLOps tools hinges on whether it creates a competitive advantage. For commodity tasks like experiment tracking, buying a managed service avoids locking up engineers.
TensorFlow Serving: A Production Server for ML Models
Think of TensorFlow Serving as a dedicated web server for your ML models. It provides a stable API for inference and manages model versions, abstracting away deployment complexity. The main footgun is thinking it only serves models; it serves any 'Servable'.
Weights & Biases: MLOps for Experiment Tracking & Evaluation
Weights & Biases is a platform for MLOps, providing experiment tracking, evaluation, and observability for AI models. It helps you develop models and ship LLM applications. The main risk it addresses is losing track of which model version used which data.
Idempotent Data Pipelines: Reruns Without Side Effects
An idempotent pipeline gives the same output for the same input, no matter how many times you run it. This lets you safely retry failed jobs without side effects, which is crucial for scheduled batch inference or feature engineering tasks.
Data Storage Tiering: Pay Only for the Access You Need
Treat data like items in a house: hot, frequently used data on the counter; cool, less-used data in the pantry. Cloud providers use this to price storage, letting you move old logs to cheaper tiers.
Stop Paying for Idle Cloud Resources
Stop paying for idle cloud servers. Automated shutdown is like turning off the lights in an empty office, running compute only when needed. It's ideal for dev environments or scheduled batch jobs. The footgun is applying this to stateful production services.
GPU Utilization: Are You Wasting Your Most Expensive Resource?
GPU utilization isn't just a percentage; it's a measure of your return on investment. It tells you if your expensive hardware is computing or just waiting for data. Use it to diagnose slow training jobs and right-size cloud instances for ML workloads.
Unit Economics: Tying ML Costs to Business Value
Unit economics connect your ML spending to business outcomes. Instead of a total cloud bill, you see cost per prediction or per token. This helps product owners make pricing tradeoffs and engineers spot efficiency gains.
Showback vs. Chargeback: Who Pays for Compute?
Showback tells teams what their resource usage costs; Chargeback makes them pay for it. It's the difference between a receipt and a bill. These models help manage cloud costs, but implementing chargeback without granular tracking leads to disputes.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles