Design a system to detect training-serving skew for a numerical feature

What's really being asked

Your ability to bridge data engineering and ML operations by designing a reliable low-latency monitoring pipeline that detects when a numerical feature's production distribution diverges from its training baseline before model quality decays. Interviewers want to see statistical rigor, operational pragmatism, and escalation design.

The full answer

First, a distribution comparison strategy rather than point checks. For a numerical feature like user_age, this means using Population Stability Index, Kolmogorov-Smirnov tests, or Jensen-Shannon divergence computed over sliding windows of inference traffic. Second, data plumbing: logging raw inference requests alongside training set snapshots, ideally via a feature store or immutable training dataset reference. Third, thresholding logic that avoids static magic numbers by using dynamic baselines or percentile-based bounds, plus burn-in periods after deployments. Fourth, an alerting taxonomy: page on-call engineers only when skew exceeds a critical threshold tied to business impact, while lower tiers trigger tickets or automated retraining pipelines. Fifth, remediation actions such as model rollback, feature flagging, or triggering an emergency retraining job with recent production data.

The mistakes people make

Comparing individual feature values between training and serving rows instead of aggregate distributions. Proposing nightly batch checks for a critical feature that needs sub-hour detection. Using fixed thresholds like plus or minus five years without considering seasonality or user base evolution. Ignoring sample size requirements and alerting on fifty requests. Suggesting manual human review as the primary detection mechanism.

What usually comes next

How would you handle categorical features differently? What if the skew is caused by a legitimate user demographic shift rather than a bug? How do you distinguish between training-serving skew and upstream data pipeline failures? What latency can you tolerate between detection and alert, and how does that drive your architecture choice?

A concrete example

At a subscription service, user_age might drift when a marketing campaign drives a wave of younger signups. The system bins age into deciles, computes PSI every fifteen minutes over a rolling one-hour window of inference traffic, and compares against the training set decile distribution. If PSI exceeds 0.25 for two consecutive windows, a low-priority ticket fires to the ML platform team. If PSI exceeds 0.4, the on-call is paged and the model automatically routes to a fallback heuristic while a retraining pipeline starts with the last twenty-four hours of labeled data.