Intermediate concepts in MLOps & Infrastructure, page 4
Why GPUs Dominate Neural Network Training
A GPU is a freight train, a CPU a race car: deep learning moves identical math across huge batches. GPUs win on transformers and CNNs. The footgun is using them for tiny models, where data transfer overhead eats the gains.
Staging Environments for ML Pipelines
A staging environment mirrors production so models and pipelines are validated on production-like data and infrastructure before release. It catches drift, integration breaks, and serving regressions early, making promotion to production a safe, repeatable…
Right-Size Inference and Stop Paying for Idle GPUs
Instance right-sizing matches inference to the smallest hardware that serves it without choking. It matters when GPU endpoints idle at 10% utilization. The footgun is copying your training spec into production; inference rarely needs that memory or multi-GPU.
Inference Health Checks: Traffic Gates, Not Heartbeats
An inference server's health check is a traffic gate, not a heartbeat. Kubernetes uses it to route requests only after the model is loaded. The footgun is probing the root path, which stays green even when the model has crashed or the GPU is wedged.
Adversarial Validation: Detect Drift with a Classifier
Adversarial validation reframes drift detection as a classification problem: train a model to separate training rows from production rows. If it can tell them apart, your feature distributions have shifted.
Directed Acyclic Graph (DAG) for Workflows
A DAG models a workflow as tasks (nodes) connected by dependency edges with no cycles, so a scheduler knows the valid execution order. It enables parallelism, safe retries, and idempotent reruns, and underpins orchestrators like Airflow for ML pipelines.
Parameterization: One Pipeline, Any Environment
Externalize every path, hyperparameter, and compute setting so one pipeline runs unchanged across dev, staging, and production. This enables reproducible experiments and safe CI/CD. The footgun is branch-per-environment repos that silently diverge.
Dynamic Fan-out/Fan-in Pipelines
Dynamic fan-out/fan-in spawns parallel tasks from runtime data, then gathers results. Use it when input counts vary, like processing a daily changing set of files. The footgun is a fan-in task that hangs waiting for branches lost to partial failure.
Centralized vs Decentralized ML Platforms
A centralized ML platform trades team autonomy for standardization, while decentralized platforms embed ML tooling inside product teams. Centralized suites drown in ticket queues; decentralized ones duplicate cost and security holes without strong governance.
Model Risk Management: The Immune System for Production Models
Model Risk Management treats every deployed model as a liability that can silently decay. Banks use it to stop bad predictions from becoming bad decisions. The footgun is treating validation as a one-time checkbox instead of continuous governance.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles