Why systematically track ML experiments and what should you log?

WHAT THIS TESTS: This question probes whether you treat experiment tracking as production infrastructure or as an afterthought. At the senior level, interviewers want to see that you understand tracking is not about creating a paper trail but about creating a reproducible, searchable record that lets a team move fast without breaking things. The core concern is operational maturity: can you debug a model six months after training, and can you onboard a teammate without losing institutional knowledge.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, name three motivations: reproducibility so you can reconstruct any past result, model selection so you can compare candidates fairly, and debugging velocity so you can isolate what changed when metrics shift. Second, list at least three logged categories: hyperparameters and configuration, because a model is a function of its parameters; metrics including both summary and per-epoch values, because they let you detect overfitting and rank candidates; and code version plus data lineage, because a result is meaningless if the training script or dataset changed underneath it. Third, explain the causal link between each log type and a business risk, such as shipping a model you cannot reproduce. Fourth, mention that tracking should be automatic and integrated into training scripts, not manual.

COMMON WRONG ANSWERS: Common wrong answers include treating tracking as optional documentation that slows you down, only logging final test accuracy without training curves, or conflating experiment tracking with model monitoring. Another red flag is listing dozens of irrelevant fields without explaining why they matter. Saying you rely on Jupyter notebook checkpoints or file names like model_final_v2.pkl signals amateur practice.

LIKELY FOLLOW-UPS: Interviewers often follow up by asking how you handle large artifacts like multi-gigabyte checkpoints, how you organize runs during distributed training across many nodes, or how you prevent sensitive hyperparameters from leaking. They may also ask how you integrate tracking with CI so that every pull request logs a baseline, or how you enforce that every production model has a corresponding experiment ID.

ONE CONCRETE EXAMPLE: Imagine you are tuning a transformer for named-entity recognition. You log the learning rate, batch size, and random seed as hyperparameters. You log training loss, validation F1, and per-entity F1 as metrics. You log the Git commit hash and data snapshot identifier as lineage. Three months later, a new release shows a five-point F1 drop on the ORG entity. Because you have per-entity metrics, you know when the regression started. Because you have the commit hash and data snapshot, you can diff the code and data to find that a preprocessing change dropped articles before company names. Without tracking, you would be guessing among unrecorded changes.