Diagnosing model degradation over time
MLOps maturity around drift.
Name it model drift, split data vs concept drift; diagnose by comparing distributions and ruling out pipeline bugs; fix via monitoring and retraining.
Blind retraining before diagnosis.
WHAT THIS TESTS Your ability to run models in production and reason about why they decay, distinguishing genuine drift from data-quality bugs, and to design monitoring and retraining rather than firefighting.
A GOOD ANSWER COVERS The phenomenon is model drift. Separate two causes: data drift, where the distribution of inputs shifts such as seasonality or a new user segment, and concept drift, where the relationship between inputs and the target changes because user behavior itself evolves. Diagnose in order. First confirm the degradation is real using monitored offline and online metrics, not noise. Then compare current input distributions against the training baseline with measures like Population Stability Index or the Kolmogorov-Smirnov test per feature. Check whether labels arrive with delay, since apparent degradation can be a measurement artifact. Rule out pipeline issues: a renamed column, a changed unit, a broken join, or a feature that silently went null often masquerades as drift. Slice metrics by segment to localize the problem.
COMMON WRONG ANSWERS Calling everything overfitting. Retraining blindly before diagnosing, which hides upstream bugs and can bake in bad data. Ignoring label latency. Having no baseline to compare against.
RETRAINING AND MONITORING Choose a strategy fit to drift speed: scheduled retraining for slow drift, trigger-based retraining when monitored metrics or PSI cross a threshold, or online learning for fast-changing signals. Validate candidates against a holdout and via shadow or canary deployment before full rollout. Monitor input distributions, prediction distributions, and ground-truth metrics with alerting.
ONE CONCRETE EXAMPLE Engagement forecasts degrade after a UI redesign changes how users browse. PSI on session-depth features spikes, signaling data drift; you retrain on recent post-redesign data, canary it to five percent of traffic, confirm metric recovery, then promote it and add a PSI alert so the next shift is caught early instead of after six months.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.