Detecting and responding to model and concept drift

WHAT THIS TESTS This checks whether you understand that a model's performance is not fixed after deployment because the world it predicts on keeps changing. It rewards concrete monitoring choices and a disciplined response process.

A GOOD ANSWER COVERS Define the terms precisely. Data drift means the distribution of incoming features changes, for example a new user demographic. Concept drift means the relationship between inputs and the target changes, for example spending behavior shifting during a recession, so the same inputs now imply different outcomes. For detection, monitor input distributions with the Population Stability Index or a Kolmogorov-Smirnov test, and monitor prediction quality directly by tracking metrics like AUC, precision, recall, or calibration error against ground-truth labels once they become available. Set thresholds that trigger alerts. Once drift is confirmed, follow a process: investigate the root cause rather than retraining blindly, collect and label recent representative data, retrain or recalibrate the model, validate it against a holdout and against the current production model, and deploy via a safe rollout such as canary, monitoring closely.

COMMON WRONG ANSWERS Treating accuracy as static and only checking it once. Monitoring only input drift while ignoring that labels may arrive late and that prediction quality is what ultimately matters. Retraining automatically on any drift signal without diagnosing whether it is real or a data-pipeline bug.

LIKELY FOLLOW-UPS What do you do when ground-truth labels are delayed by weeks? How do you distinguish drift from a broken upstream pipeline? How often should you retrain?

ONE CONCRETE EXAMPLE A credit-risk model's PSI on income jumps above 0.25, signaling input drift, and a month later its AUC on newly defaulted accounts falls from 0.82 to 0.74. You confirm it is a genuine economic shift, retrain on the last six months of labeled data, validate the new model beats the old on a recent holdout, and canary it to ten percent of traffic before full rollout.