How would you design a reproducible ML training pipeline?

WHAT THIS TESTS: This question tests whether you treat ML training as a software engineering problem rather than an ad-hoc experiment. Interviewers want to see that you understand reproducibility requires immutability across three distinct planes: the training code, the input data, and the execution environment. They are also checking if you know how to automate these steps rather than relying on manual processes.

A GOOD ANSWER COVERS: First, code versioning with Git and explicit commit hashes referenced in every training run metadata so the exact code state is always recoverable. Second, data versioning using tools like DVC, lakehouse table formats like Iceberg or Delta Lake, or at minimum immutable blob storage with content-addressable hashes so dataset lineage is preserved. Third, environment reproducibility via containerization with Docker and locked dependency files such as poetry.lock or requirements.txt with strict hashes to eliminate drift in libraries and system packages. Fourth, pipeline orchestration through a CI/CD system or workflow engine like Kubeflow Pipelines or GitHub Actions that triggers training automatically on code or data changes and stores artifacts with unique run IDs. Fifth, experiment tracking with a tool like MLflow or Weights and Biases that logs hyperparameters, metrics, and the exact versions of code, data, and containers used.

COMMON WRONG ANSWERS: Saying you version control Jupyter notebooks as the primary source of truth without converting to modular Python scripts. Claiming that Git LFS alone solves data versioning for terabyte-scale datasets. Proposing manual training runs on a local laptop without containerization or dependency pinning. Ignoring the need to version the training script itself separately from the model artifact.

LIKELY FOLLOW-UPS: How would you reproduce a model trained six months ago if the original data source has been overwritten? How do you handle non-determinism from GPU operations or random seeds across different hardware? At what scale would you move from DVC to a dedicated feature store or data lakehouse? How do you verify that a reproduced training run actually produced the same model weights?

ONE CONCRETE EXAMPLE: A senior candidate might describe a pipeline where a GitHub Actions workflow triggers on a pull request. The workflow builds a Docker image from a pinned base CUDA image and installs dependencies from a poetry.lock file. It then pulls a dataset version tagged by a DVC pointer file stored in Git, trains the model while logging the Git commit SHA and DVC data hash to MLflow, and finally pushes the model artifact and container image to a registry tagged with the run ID. If a bug is found in production, the team can check out the exact commit, pull the same container image by digest, and re-run training with identical data to reproduce the issue within hours instead of weeks.

Source: ml-ops.org