Intermediate everything in MLOps & Infrastructure, page 4
Design training job submission to a shared Kubernetes cluster
Gateway with artifact caching; namespace quotas; GPU schedulers like Volcano; Prometheus metrics and cost attribution.
Robust checkpointing strategy for multi-day training jobs and seamless resumption
Tests production-grade distributed training reliability. Cover async atomic checkpoints, MTBF-based cadence, tiered storage, and recovery drills. Red flag: blocking synchronous writes that ignore silent corruption or straggler finalization.
How do you version and distribute Docker dev environments consistently?
Tests immutable dev environment distribution. Strong answers cover: versioned Dockerfiles in Git, immutable image tags pushed to a registry, and enforcing identical pulls for CI and developers. Red flag: using the "latest" tag or local Dockerfile rebuilds.
How do you build dev and production Docker images from one source?
Mastery of Docker multi-stage builds for isolating dev and production dependencies. Use a base stage, a dev target with linters and tests, and a lean production target copying only the build artifact.
How do you persist notebooks and artifacts in Docker?
Tests Docker storage abstractions. A strong answer distinguishes bind mounts for live notebook editing from named volumes for datasets and artifacts, and warns against docker commit for persistence. Red flag: treating containers as stateful VMs.
Design a CI/CD pipeline that automates model promotion from Staging to Production
Tests whether you treat model promotion as a gated software delivery workflow. Strong answers use registry state-change triggers, automated drift and performance checks, canary deployment gates, and rollback.
Why use a Model Registry over dated pickle files?
A strong answer covers versioning, lineage, promotion aliases, and governance.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion.
Design a system to detect training-serving skew for a numerical feature
Tests ML monitoring design via statistical distribution comparison between training and live data. Strong answers cover PSI/KS tests, windowed thresholding, and tiered alerting. Red flag: comparing raw values instead of distributions or ignoring alert fatigue.
How would you design automatic data drift detection for production inference?
This tests reference-vs-live monitoring architecture. A strong answer chunks data, runs univariate per-feature drift, adds multivariate PCA or domain classifiers, and ranks threshold alerts. A red flag is checking aggregate metrics instead of feature shifts.
How would you scale 1TB Pandas feature computation across machines?
This tests memory limits and distributed migration. A strong answer contrasts single-machine tactics, column pruning and efficient dtypes, with distributed frameworks like Dask or Spark, noting shuffle costs and API parity.
Describe feature store architecture and training-serving skew
This tests FTI pipeline glue and dual-store skew elimination. A strong answer lists offline and online stores, shared transformation logic for consistent compute, and point-in-time correctness. A red flag is calling it merely a database or cache.
What production metrics and auto-thresholds trigger model retraining?
This tests production monitoring maturity. A strong answer covers technical drift metrics, business KPIs tied to model decisions, and automated thresholds that page or trigger CI/CD retraining.
Explain ML pipelines and typical CI/CD/CT components
Tests if you separate code CI/CD from model CT and grasp ML automation. Cover source control, build, tests, deploy for code; data validation, training, evaluation, promotion for CT. Red flag: treating ML like software CI/CD and ignoring data or registry gates.
Why version code, data, and models in MLOps?
Tests immutable lineage across code, data, and models. Strong answers cover content-addressed data, git commits, a model registry linking both, and CI triggers on any change. Red flag: saying git alone handles data and models.
Model Risk Management: The Immune System for Production Models
Model Risk Management treats every deployed model as a liability that can silently decay. Banks use it to stop bad predictions from becoming bad decisions. The footgun is treating validation as a one-time checkbox instead of continuous governance.
Centralized vs Decentralized ML Platforms
A centralized ML platform trades team autonomy for standardization, while decentralized platforms embed ML tooling inside product teams. Centralized suites drown in ticket queues; decentralized ones duplicate cost and security holes without strong governance.
Dynamic Fan-out/Fan-in Pipelines
Dynamic fan-out/fan-in spawns parallel tasks from runtime data, then gathers results. Use it when input counts vary, like processing a daily changing set of files. The footgun is a fan-in task that hangs waiting for branches lost to partial failure.
Parameterization: One Pipeline, Any Environment
Externalize every path, hyperparameter, and compute setting so one pipeline runs unchanged across dev, staging, and production. This enables reproducible experiments and safe CI/CD. The footgun is branch-per-environment repos that silently diverge.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles