Intermediate concepts in MLOps & Infrastructure, page 3
Argo Workflows: Run Complex Jobs on Kubernetes
Think of Argo Workflows as a script runner for Kubernetes, where each command is a container. It runs multi-step jobs like CI/CD or ML pipelines. The footgun is treating it like a full CI server; it's just an engine and lacks features like Git polling.
Pipeline Step Caching: Don't Recompute What You Don't Have To
Pipeline step caching is memoization for your ML infrastructure, saving time and money by reusing previous results. It's used in MLOps pipelines when inputs and code haven't changed. The footgun: the cache is scoped to one pipeline and a timeout, not globally.
The MLOps Maturity Model: A Roadmap for Growth
The MLOps Maturity Model is a roadmap from manual chaos to automated ML systems. Use it to assess your team's current state and plan incremental improvements.
ML Threat Modeling: Assume Your Data Is Compromised
Threat modeling for ML means assuming your training data is already compromised. This is crucial for services using public or user-supplied datasets. The main footgun is trusting data sources, as data poisoning can silently corrupt your model's behavior.
SHAP: Explaining Black Box Model Predictions
SHAP explains a model's prediction by treating features as players in a game and fairly distributing credit for the outcome. Use it to understand why a specific loan was denied or an image was misclassified. The footgun: SHAP explains the model, not reality.
LIME: Explaining Single Predictions from Any ML Model
LIME explains a single prediction from any 'black box' model by approximating it with a simpler model that's only accurate locally. Use it to see why a specific user churned.
Showback vs. Chargeback: Who Pays for Compute?
Showback tells teams what their resource usage costs; Chargeback makes them pay for it. It's the difference between a receipt and a bill. These models help manage cloud costs, but implementing chargeback without granular tracking leads to disputes.
Unit Economics: Tying ML Costs to Business Value
Unit economics connect your ML spending to business outcomes. Instead of a total cloud bill, you see cost per prediction or per token. This helps product owners make pricing tradeoffs and engineers spot efficiency gains.
GPU Utilization: Are You Wasting Your Most Expensive Resource?
GPU utilization isn't just a percentage; it's a measure of your return on investment. It tells you if your expensive hardware is computing or just waiting for data. Use it to diagnose slow training jobs and right-size cloud instances for ML workloads.
Stop Paying for Idle Cloud Resources
Stop paying for idle cloud servers. Automated shutdown is like turning off the lights in an empty office, running compute only when needed. It's ideal for dev environments or scheduled batch jobs. The footgun is applying this to stateful production services.
Data Storage Tiering: Pay Only for the Access You Need
Treat data like items in a house: hot, frequently used data on the counter; cool, less-used data in the pantry. Cloud providers use this to price storage, letting you move old logs to cheaper tiers.
Idempotent Data Pipelines: Reruns Without Side Effects
An idempotent pipeline gives the same output for the same input, no matter how many times you run it. This lets you safely retry failed jobs without side effects, which is crucial for scheduled batch inference or feature engineering tasks.
Weights & Biases: MLOps for Experiment Tracking & Evaluation
Weights & Biases is a platform for MLOps, providing experiment tracking, evaluation, and observability for AI models. It helps you develop models and ship LLM applications. The main risk it addresses is losing track of which model version used which data.
TensorFlow Serving: A Production Server for ML Models
Think of TensorFlow Serving as a dedicated web server for your ML models. It provides a stable API for inference and manages model versions, abstracting away deployment complexity. The main footgun is thinking it only serves models; it serves any 'Servable'.
MLOps: When to Build vs. Buy Your Infrastructure
Deciding to build or buy MLOps tools hinges on whether it creates a competitive advantage. For commodity tasks like experiment tracking, buying a managed service avoids locking up engineers.
Model Registry: Source of Truth for Deployed Models
A model registry is the source of truth for which trained model runs where, turning anonymous artifact files into versioned, staged assets. It matters when you deploy multiple models or need instant rollbacks.
Log Transformation: Compress the Long Tail
Log transformation compresses the long tail of skewed data so outliers cannot dominate loss. Use it for features like income or latency that span orders of magnitude. The footgun is blindly applying it to zeros or negatives, which destroys data.
Experiment Run: The Immutable Training Receipt
An experiment run is an auto-generated log for one training job: it captures hyperparameters, metrics, code, and artifacts. Teams use runs to debug regressions and audit settings. The footgun is logging many metrics without versioning data so comparison fails.
MLflow Models Standardize Deployment Packaging
MLflow Models wrap artifacts into a standard package so one pipeline serves sklearn or PyTorch without new deployment code. Teams ship experiments to REST endpoints without Dockerfiles per model. Missing dependency logging lets model load but fail to predict.
Docker Image vs. Container: Blueprint vs. Runtime
A Docker image is a read-only blueprint; a container is a live instance with a writable layer. You build an image once in CI and run many containers from it in production. The footgun is mutating a running container without updating the image recipe.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles