How do you diagnose and fix a model's degrading performance?

What's really being asked

This question tests your MLOps maturity and structured problem-solving skills. The interviewer is looking for more than the definition of "concept drift"; they want to see if you can own the entire lifecycle of a production model, from diagnosing failure to implementing a robust, long-term solution.

The full answer

Your answer should have four distinct parts. First, IDENTIFY the phenomenon as concept drift, where the statistical properties of the target variable change over time. Second, propose a SYSTEMATIC DIAGNOSIS. This involves quantifying the performance drop (e.g., AUC from 0.85 to 0.72), then checking for data drift by comparing feature distributions (e.g., using Population Stability Index) between training and live data. Then, investigate true concept drift by analyzing how feature-target relationships have changed over time. Third, outline a SOLUTION. This is not just "retrain." Specify a retraining strategy based on the diagnosis, such as using a 90-day rolling window of data or adding new features to capture new user behaviors. Fourth, describe a PROACTIVE MONITORING strategy. This means setting up automated alerts on data distribution shifts (e.g., PSI > 0.2) and model accuracy degradation (e.g., >5% drop week-over-week).

The mistakes people make

The most common red flag is jumping immediately to the solution: "We should retrain the model." This demonstrates a lack of diagnostic rigor. A senior engineer investigates why before acting. Another weak answer is being vague about monitoring, saying "we'll watch the performance" without specifying metrics (AUC, F1-score), statistical tests (KS-test, PSI), or concrete alerting thresholds. Finally, failing to distinguish between data drift (input distributions change) and concept drift (feature-target relationships change) can be a sign of a more junior understanding.

What usually comes next

Expect questions like: "How would you set the threshold for your drift detection alerts?", "What's the difference between concept drift and covariate shift?", "Describe the infrastructure you'd use to implement this monitoring system.", or "What if retraining the model doesn't fix the problem?"

A concrete example

A user engagement model's AUC dropped from 0.85 to 0.72 in six months. Diagnosis revealed a new "short-form video" feature was launched. The distribution of the session_duration_minutes feature shifted dramatically towards shorter sessions. More importantly, the concept drifted: previously, long sessions correlated with high engagement, but now many short, high-frequency sessions also did. The solution was to retrain the model on data from the last 3 months and add a new feature, uses_short_form_video, to capture the new behavior pattern.