How do you monitor model health with delayed ground truth labels?

What's really being asked

This question probes production ML operations maturity beyond standard accuracy and loss curves. Interviewers want to see if you can design a monitoring stack that functions when the primary feedback loop is broken by weeks or months of label delay. The core concepts are statistical drift detection, proxy metric design, and human evaluation pipelines.

The full answer

First, upstream input monitoring. You should detect training-serving skew and data drift using statistical tests like Population Stability Index or Kolmogorov-Smirnov tests on key features, with PSI alert thresholds commonly set between 0.1 and 0.25 depending on criticality. Second, model output monitoring. Track prediction distribution shifts, confidence score calibration, and entropy changes; if predicted class distributions move more than two to five percent from baseline, that warrants investigation. Third, proxy and business metrics. Use intermediate signals such as click-through rates, conversion events, or human escalation rates as leading indicators of model quality, validating their correlation with ground truth historically. Fourth, human-in-the-loop evaluation. Implement spot-checking or shadow labeling on a stratified sample of predictions to estimate precision and recall before full labels arrive. Fifth, safe deployment patterns. Use champion-challenger setups or shadow mode so new models run in parallel without user impact, since you cannot immediately validate them against delayed labels.

The mistakes people make

Waiting passively for ground truth without interim monitoring. Proposing automated retraining on fresh data without explaining how you validate the new model when labels are delayed. Relying solely on aggregate business metrics without connecting them back to model drift. Confusing data drift with concept drift and applying the wrong statistical test. Suggesting synthetic labels as a primary monitoring strategy without discussing their validation.

What usually comes next

How do you disentangle seasonality from true data drift in your feature distributions? What PSI threshold would trigger a page versus a ticket? How do you prove your proxy metric actually correlates with the delayed ground truth? What is your rollback strategy if drift is detected but accuracy remains unknown for another month? How do you handle class imbalance when your human spot-check sample is small?

A concrete example

Consider a fraud model where chargeback labels arrive sixty days later. You would monitor transaction feature distributions hourly with PSI, paging if merchant category distributions shift above 0.2 PSI. You would track the model's score distribution for sudden spikes in low-confidence predictions. You would run a daily human review sample of five hundred flagged transactions to estimate precision. You would also compare auto-approval rates against historical baselines; if auto-approval rates jump fifteen percent while downstream chargeback proxies stay flat, that signals potential model decay weeks before labels confirm it.