How to establish data lineage and reproducibility for hundreds of ML models

WHAT THIS TESTS: Whether you can design a unified lineage system that traces every ML artifact from raw data source through transformations, training, and deployment. The interviewer wants to see you treat lineage as a graph problem requiring automated metadata capture, not as a documentation chore.

A GOOD ANSWER COVERS: First, data origin tracking by pinpointing sources such as databases, APIs, or file systems and assigning immutable dataset IDs with versioning. Second, transformation lineage by recording every processing step including cleaning, feature engineering, and aggregation so that input and output schemas are fully auditable. Third, model training lineage by capturing the exact data version, code commit, container image, hyperparameters, and resulting model artifacts in a system like MLflow. Fourth, deployment lineage by tracking the deployment environment, model version, and timeline so that production endpoints link back to training runs. Fifth, a unified query interface that exposes these relationships as a directed graph for upstream and downstream traversal.

COMMON WRONG ANSWERS: Proposing manual documentation in wikis or spreadsheets. Suggesting each team picks its own tool without a unified schema. Tracking only the final training dataset while ignoring upstream feature transformations. Storing lineage metadata separately from the model registry so the two drift apart. Focusing only on data versioning and neglecting code, environment, and model artifact links.

LIKELY FOLLOW-UPS: How would you handle lineage for streaming features versus batch features? What is your retention and eviction policy for lineage metadata when you have millions of runs? How do you ensure lineage capture does not add more than five to ten percent overhead to training jobs? How would you use lineage to perform impact analysis before deprecating a raw data source?

ONE CONCRETE EXAMPLE: A data scientist notices a drop in model accuracy for a fraud detection model. Using the lineage graph in MLflow, she queries the model and sees it was trained on dataset version 4.2. She traces back to find that version 4.2 was generated by a feature pipeline that started consuming a new API source two weeks ago. The pipeline log shows a schema change in a timestamp field that caused a feature transformation to silently fail. Because lineage linked the deployed model, training run, dataset version, transformation code, and raw source, the root cause was identified in minutes rather than days, demonstrating accelerated debugging and reproducibility.