How would you design a reproducible ML training pipeline?
Tests if you can version ML's three moving parts: code, data, and environment. Good answers cover Git for code, DVC or lakehouse versioning for data, and Docker plus locked dependencies for environments.
What's really being asked
This question tests whether you treat ML training as a software engineering problem rather than an ad-hoc experiment. Interviewers want to see that you understand reproducibility requires immutability across three distinct planes: the training code, the input data, and the execution environment. They are also checking if you know how to automate these steps rather than relying on manual processes.
The full answer
First, code versioning with Git and explicit commit hashes referenced in every training run metadata so the exact code state is always recoverable. Second, data versioning using tools like DVC, lakehouse table formats like Iceberg or Delta Lake, or at minimum immutable blob storage with content-addressable hashes so dataset lineage is preserved. Third, environment reproducibility via containerization with Docker and locked dependency files such as poetry.lock or requirements.txt with strict hashes to eliminate drift in libraries and system packages. Fourth, pipeline orchestration through a CI/CD system or workflow engine like Kubeflow Pipelines or GitHub Actions that triggers training automatically on code or data changes and stores artifacts with unique run IDs. Fifth, experiment tracking with a tool like MLflow or Weights and Biases that logs hyperparameters, metrics, and the exact versions of code, data, and containers used.
The mistakes people make
Saying you version control Jupyter notebooks as the primary source of truth without converting to modular Python scripts. Claiming that Git LFS alone solves data versioning for terabyte-scale datasets. Proposing manual training runs on a local laptop without containerization or dependency pinning. Ignoring the need to version the training script itself separately from the model artifact.
What usually comes next
How would you reproduce a model trained six months ago if the original data source has been overwritten? How do you handle non-determinism from GPU operations or random seeds across different hardware? At what scale would you move from DVC to a dedicated feature store or data lakehouse? How do you verify that a reproduced training run actually produced the same model weights?
A concrete example
A senior candidate might describe a pipeline where a GitHub Actions workflow triggers on a pull request. The workflow builds a Docker image from a pinned base CUDA image and installs dependencies from a poetry.lock file. It then pulls a dataset version tagged by a DVC pointer file stored in Git, trains the model while logging the Git commit SHA and DVC data hash to MLflow, and finally pushes the model artifact and container image to a registry tagged with the run ID. If a bug is found in production, the team can check out the exact commit, pull the same container image by digest, and re-run training with identical data to reproduce the issue within hours instead of weeks.
Interview question
To reproduce a model six months later after the original data source was overwritten, what must the pipeline have captured at training time?
- a.The Jupyter notebook checkpoint and a timestamp of when training started
- b.Only the Git commit hash and the random seed used for initialization
- c.The Git commit hash, a content-addressable data version, and the exact container image digestCorrect
- d.The MLflow run ID and a requirements.txt without strict hashes
Why? this is the answer
Full reproducibility requires immutable snapshots of code, data, and environment, so all three must be captured. Option B is tempting because versioned code and random seeds feel sufficient, but they cannot recover a dataset that has been overwritten or guarantee the same system environment.
Just read this? Test yourself on what you have been reading.
Read the original → ml-ops.org
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles