Mlops
265 bites tagged Mlops — interview questions with model answers, and 60-second explainers.
What events trigger automatic model retraining beyond code changes?
This tests whether you treat ML pipelines as event-driven systems, not just software CI/CD. A strong answer lists data drift, scheduled cron jobs, production metric degradation, schema changes, and upstream data pipeline completion.
What is a model registry's purpose in CI/CD4ML and its CI/CD interaction?
Tests if you see the model registry as the bridge between experiments and production, not just storage. A strong answer explains how CI publishes validated artifacts and CD consumes versioned models. Red flag: calling it a passive file dump without versioning.
What automated tests belong in CI before deploying a classification model?
Name data schema checks, performance regression vs baseline, bias audits, and artifact integrity. Distinguishing code tests from ML-specific CI validation. Only testing the inference API while ignoring model behavior.
Key differences between traditional and ML CI/CD pipelines?
Contrast code deploys with data versioning, model registries, and retraining; note holdout eval. ML CI/CD manages data and model lineage, not just code. Treating the model as a static binary ignoring data or retraining context.
Design multi-tenant GPU cluster scheduling and preemption policies
Tests ability to design fair GPU scheduling preventing starvation and noisy-neighbor issues. Answer: Kueue for fair-share, namespace quotas with MIG, priority classes with backoff.
Compare Airflow and Kubeflow for ML training pipelines
Tests orchestrator-to-workload fit. Strong answers contrast Airflow's data integration and Python DAGs with Kubeflow's K8s scaling, container reproducibility, and experiment tracking. Red flag: claiming one is always better without stage-specific reasoning.
Design training job submission to a shared Kubernetes cluster
Gateway with artifact caching; namespace quotas; GPU schedulers like Volcano; Prometheus metrics and cost attribution. Multi-tenant ML infrastructure with usability, fairness, observability.
Robust checkpointing strategy for multi-day training jobs and seamless resumption
Tests production-grade distributed training reliability. Cover async atomic checkpoints, MTBF-based cadence, tiered storage, and recovery drills. Red flag: blocking synchronous writes that ignore silent corruption or straggler finalization.
Difference between data and model parallelism, and when to prefer each
Tests split axis: data parallelism replicates model and shards data; model parallelism shards model across devices. Use data parallelism for throughput; model parallelism when layers exceed GPU memory.
How do you containerize a Python training script for GPU cloud VMs?
This tests reproducible GPU containerization. A strong answer uses an NVIDIA CUDA base image, installs Python dependencies at build time, copies the training script, and runs with --gpus.
Design on-demand containerized dev environments for data scientists
Tests multi-tenant notebook infrastructure design. Cover a Notebook Controller, curated Jupyter and VS Code images, namespace isolation with RBAC, resource quotas, and persistent storage. Red flag: a single shared VM without tenancy or idle shutdown.
Strategies to reduce a 5GB ML Docker image size
Tests multi-stage build hygiene and ML bloat reduction. Strong answers use multi-stage builds, strip CUDA dev libs, use slim bases, and collapse cache cleanup into one RUN. Red flag: rm -rf in a separate RUN step, which still bloats the layer.
Describe two secure methods for providing secrets to a running container
Mention runtime mounts like Docker secrets, orchestrator secret injection, and cloud IAM patterns. Containerized ML security, avoiding baked-in credentials. Proposing .env files baked into layers or committed to git.
How do you build dev and production Docker images from one source?
Mastery of Docker multi-stage builds for isolating dev and production dependencies. Use a base stage, a dev target with linters and tests, and a lean production target copying only the build artifact. Separate Dockerfiles or dev tools in production.
How do you persist notebooks and artifacts in Docker?
Tests Docker storage abstractions. A strong answer distinguishes bind mounts for live notebook editing from named volumes for datasets and artifacts, and warns against docker commit for persistence. Red flag: treating containers as stateful VMs.
How do you structure a Dockerfile to leverage layer caching for dependencies?
Tests Docker layer invalidation and cache-aware instruction ordering. Copy requirements.txt and run pip install before source code so deps cache independently. Red flag: copying everything at once or installing deps after code, busting cache every build.
Walk me through essential Dockerfile commands for a reproducible Python ML environment
Tests your ability to containerize Python ML scripts reproducibly. A strong answer covers FROM with a pinned slim image, WORKDIR, COPY for requirements and code, RUN pip install, and CMD or ENTRYPOINT.
How would you design a system to detect training-serving skew using model registry metadata?
This tests statistical monitoring between production data and registry training baselines. Strong answers: schema-bound metadata, incremental stats, drift metrics PSI, tiered alerting. Red flag: schema validation mistaken for drift or manual checks only.
Design a CI/CD pipeline that automates model promotion from Staging to Production
Tests whether you treat model promotion as a gated software delivery workflow. Strong answers use registry state-change triggers, automated drift and performance checks, canary deployment gates, and rollback.
Why use a Model Registry over dated pickle files?
A strong answer covers versioning, lineage, promotion aliases, and governance. Why ad-hoc storage fails as models and teams scale. Treating deployment as a file copy rather than a managed lifecycle.
Argue for declarative or imperative feature platforms with trade-offs
This tests whether you weigh control flow against data flow. A strong answer argues from org maturity: declarative systems abstract DAG topology, while imperative ones offer Spark control at the cost of manual idempotency. Red flag: ignoring org culture.
Design system ensuring point-in-time correctness for training data joins
Tests temporal join design to prevent data leakage from slowly changing dimensions. Strong answers use an AS OF join on entity ID and timestamp, materialize features as of label time, and handle late arrivals. Joining on user_id alone is a red flag.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion. Isolating large-scale backfill from production while ensuring correctness.
Get Mlops bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.