Intermediate interview questions in MLOps & Infrastructure
Why version code, data, and models in MLOps?
Tests immutable lineage across code, data, and models. Strong answers cover content-addressed data, git commits, a model registry linking both, and CI triggers on any change. Red flag: saying git alone handles data and models.
Explain ML pipelines and typical CI/CD/CT components
Tests if you separate code CI/CD from model CT and grasp ML automation. Cover source control, build, tests, deploy for code; data validation, training, evaluation, promotion for CT. Red flag: treating ML like software CI/CD and ignoring data or registry gates.
What production metrics and auto-thresholds trigger model retraining?
This tests production monitoring maturity. A strong answer covers technical drift metrics, business KPIs tied to model decisions, and automated thresholds that page or trigger CI/CD retraining.
Versioning a 10TB dataset as code
Treat data like code via content-addressed pointers in Git while bytes live in object storage; dedupe by hashing so versions share unchanged files.
Describe feature store architecture and training-serving skew
This tests FTI pipeline glue and dual-store skew elimination. A strong answer lists offline and online stores, shared transformation logic for consistent compute, and point-in-time correctness. A red flag is calling it merely a database or cache.
How would you scale 1TB Pandas feature computation across machines?
This tests memory limits and distributed migration. A strong answer contrasts single-machine tactics, column pruning and efficient dtypes, with distributed frameworks like Dask or Spark, noting shuffle costs and API parity.
How would you design automatic data drift detection for production inference?
This tests reference-vs-live monitoring architecture. A strong answer chunks data, runs univariate per-feature drift, adds multivariate PCA or domain classifiers, and ranks threshold alerts. A red flag is checking aggregate metrics instead of feature shifts.
Design a system to detect training-serving skew for a numerical feature
Tests ML monitoring design via statistical distribution comparison between training and live data. Strong answers cover PSI/KS tests, windowed thresholding, and tiered alerting. Red flag: comparing raw values instead of distributions or ignoring alert fatigue.
Sub-20ms online feature serving
An in-memory key-value store (Redis) as the online feature store, precomputed features, streaming updates, and offline-online consistency.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Why use a Model Registry over dated pickle files?
A strong answer covers versioning, lineage, promotion aliases, and governance.
Design a CI/CD pipeline that automates model promotion from Staging to Production
Tests whether you treat model promotion as a gated software delivery workflow. Strong answers use registry state-change triggers, automated drift and performance checks, canary deployment gates, and rollback.
How do you persist notebooks and artifacts in Docker?
Tests Docker storage abstractions. A strong answer distinguishes bind mounts for live notebook editing from named volumes for datasets and artifacts, and warns against docker commit for persistence. Red flag: treating containers as stateful VMs.
How do you build dev and production Docker images from one source?
Mastery of Docker multi-stage builds for isolating dev and production dependencies. Use a base stage, a dev target with linters and tests, and a lean production target copying only the build artifact.
How do you version and distribute Docker dev environments consistently?
Tests immutable dev environment distribution. Strong answers cover: versioned Dockerfiles in Git, immutable image tags pushed to a registry, and enforcing identical pulls for CI and developers. Red flag: using the "latest" tag or local Dockerfile rebuilds.
Diagnosing poor distributed training scaling
Communication overhead (gradient all-reduce, interconnect), data-loading starvation, load imbalance, and small per-GPU batches; profile with the PyTorch profiler and Nsight.
Robust checkpointing strategy for multi-day training jobs and seamless resumption
Tests production-grade distributed training reliability. Cover async atomic checkpoints, MTBF-based cadence, tiered storage, and recovery drills. Red flag: blocking synchronous writes that ignore silent corruption or straggler finalization.
Design training job submission to a shared Kubernetes cluster
Gateway with artifact caching; namespace quotas; GPU schedulers like Volcano; Prometheus metrics and cost attribution.
Compare Airflow and Kubeflow for ML training pipelines
Tests orchestrator-to-workload fit. Strong answers contrast Airflow's data integration and Python DAGs with Kubeflow's K8s scaling, container reproducibility, and experiment tracking. Red flag: claiming one is always better without stage-specific reasoning.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles