How do you handle model performance degradation over time?

What's really being asked

This question tests your ability to manage the full lifecycle of a model in production. It's not about defining a term, but about demonstrating a systematic, data-driven process for diagnosing, fixing, and preventing model degradation. Interviewers want to see MLOps maturity and a proactive mindset, not just modeling skill.

The full answer

Four key steps in order. First, correctly identify the phenomenon as concept drift, defining it as a change in the statistical properties of the target variable over time, which invalidates the model. Second, propose a systematic diagnosis before acting: analyze input feature distributions (data drift), check for changes in the target variable's distribution, and analyze the model's errors to find patterns (e.g., are errors clustered in a new user segment?). Third, discuss solutions, primarily retraining, but with specific strategies like using a sliding window of recent data (e.g., the last 90 days) or weighting recent data more heavily. Fourth, outline a proactive monitoring and alerting strategy. This includes tracking data distributions (e.g., with KS tests) and key model metrics (e.g., AUC, precision), with automated alerts for significant drops (e.g., a 5% decrease).

The mistakes people make

The biggest red flag is immediately saying "just retrain the model." This lacks diagnostic rigor and might not solve the underlying problem if, for example, an upstream data pipeline is broken. Another mistake is conflating concept drift (relationship between features and target changes) with data drift (input feature distributions change) without nuance. A weak answer also lacks a forward-looking plan for monitoring, suggesting a reactive rather than proactive mindset. Finally, failing to mention analyzing the model's errors is a missed opportunity to show deep diagnostic skill.

What usually comes next

"How would you set the threshold for your monitoring alerts? What's the trade-off?" "What if retraining doesn't fix the problem? What else could be the cause?" "Describe the infrastructure you'd need to implement this monitoring and retraining pipeline." "How do you differentiate between true concept drift and short-term seasonality?"

A concrete example

For a user engagement model, concept drift might occur if a new product feature fundamentally changes what "engagement" means. For instance, adding a "Stories" feature might shift engagement from "likes" to "views." Diagnosis would involve seeing the distribution of the target variable (e.g., daily engagement score) shift significantly. The solution would be to retrain the model on data collected after the feature launch, possibly with new features related to "Stories." Monitoring would track the distribution of engagement scores and alert if it deviates by more than two standard deviations from the training distribution.