Debug sudden model degradation using experiment tracking and model registry

WHAT THIS TESTS: Whether the candidate understands the operational difference between experiment tracking, which captures training-time metadata, and the model registry, which governs deployment-time artifacts. The interviewer cares if you can construct a causal chain from training configuration to production behavior instead of guessing.

A GOOD ANSWER COVERS: First, check the model registry to confirm which exact artifact is running in production, including its version hash, stage label, and deployment timestamp, to rule out an accidental promotion of the wrong model. Second, pull the experiment run linked to that artifact to inspect training hyperparameters, dataset version or hash, feature engineering code version, and metric curves, looking for a recent change that correlates with the degradation onset. Third, compare the production input distribution against the training dataset snapshot logged in the experiment tracker to identify data drift or schema changes. Fourth, verify that the serving infrastructure is loading the same serialization format and dependencies recorded in the model registry environment metadata, because a library version mismatch can silently corrupt predictions. Fifth, trace the lineage backward through registered artifacts to see if an upstream data pipeline change produced a new training set that was inadvertently consumed.

COMMON WRONG ANSWERS: Treating the experiment tracker as a metrics dashboard rather than a lineage tool. Blaming data drift without ever checking which model version is actually deployed. Suggesting a full retrain before comparing the current production artifact against the last known good version in the registry. Ignoring dependency or schema metadata and focusing only on accuracy numbers.

LIKELY FOLLOW-UPS: How would you automate this investigation into an alert-driven playbook. What would you do if the experiment logs showed no changes but the registry proved the wrong model was promoted. How do you handle a case where the training data snapshot was not versioned.

ONE CONCRETE EXAMPLE: Imagine a computer vision model deployed for defect detection suddenly drops precision by twelve percent. In the model registry, the candidate finds that version 2.3.1 was promoted three days ago, replacing 2.2.4. The linked experiment run shows that 2.3.1 used a new augmentation pipeline and a resized input resolution of 224 by 224, whereas 2.2.4 used 512 by 512. The production logs reveal that the resizing interpolation method in the serving container defaulted to nearest-neighbor instead of bilinear, introducing aliasing that the training artifact never encountered. The candidate rolls back to 2.2.4 via the registry and files a ticket to pin the preprocessing library version in the model registry metadata.