Detecting data drift on a continuous feature

WHY IT EXISTS Models assume that production data looks like training data. When the real-world input distribution shifts, for example user behavior, sensors, or upstream pipelines change, the model's assumptions break and predictions degrade silently because the code still runs fine. Monitoring for data drift catches this before accuracy quietly erodes.

WHAT IT IS Data drift, also called covariate shift, is a change in the distribution of the input features between the training reference and the live serving data. It is distinct from concept drift, where the relationship between inputs and the target changes, and from label drift, a change in the target distribution.

A TEST FOR A CONTINUOUS FEATURE The Kolmogorov-Smirnov two-sample test is a common choice. It compares the empirical cumulative distribution functions of a recent production sample and a training reference sample and returns a statistic and a p-value. Population Stability Index is another common metric.

HOW TO INTERPRET IT The null hypothesis is that both samples come from the same distribution. A small p-value, below your chosen significance level such as 0.05, rejects the null and indicates the production distribution has drifted significantly; a large p-value gives no evidence of drift. In monitoring you run this per feature on a schedule and alert when p-values fall below threshold, then investigate or trigger retraining.

WHEN IT MATTERS It matters for any long-lived production model, especially where upstream data sources or user populations change over time.

ONE CONCRETE EXAMPLE A credit model's income feature was trained on pre-recession data. A weekly KS test against the training reference returns a p-value of 0.001, far below 0.05, flagging that the income distribution has shifted. The team is alerted, confirms a real population change, and schedules retraining before the model's calibration drifts further.