How would you design automatic data drift detection for production inference?

WHAT THIS TESTS: The interviewer wants to see if you can operationalize drift detection as a structured pipeline rather than a one-off script. They care about separation of concerns: data ingestion, period management, statistical comparison, and alerting. The question also checks whether you understand the difference between univariate feature-level drift and multivariate covariate shift, and how to handle both categorical and numerical columns without conflating them.

A GOOD ANSWER COVERS: First, establish a reference period from training or validation data and an analysis period from live inference traffic. Second, chunk the analysis stream using time-based or size-based chunking so comparisons are statistically meaningful. Third, run univariate drift detection on every individual feature, treating numerical and categorical columns separately by comparing distributions within each chunk against the reference. Fourth, add multivariate drift detection such as PCA data reconstruction or a domain classifier to catch interactions that univariate tests miss. Fifth, apply configurable thresholds, either constant or standard deviation based, to trigger alerts. Sixth, rank alerts by alert count or correlation ranking so operators know which features to investigate first. Seventh, include data quality checks for missing values and unseen values because these often precede drift.

COMMON WRONG ANSWERS: A major red flag is suggesting you only need to monitor overall standard performance metrics; that catches impact but not root cause and lags behind the actual shift. Another mistake is proposing to compare every single incoming row against the full training set; this is computationally infeasible and statistically noisy. Candidates also err by wanting a single test for both numerical and categorical features; the reference treats these separately in univariate detection. Finally, ignoring chunking and thresholds leads to alert fatigue or missed slow drifts.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle delayed ground truth and whether you would use performance estimation like CBPE or DLE to estimate drift impact before labels arrive. They might also ask how you would store and version reference datasets, or how you would decide when to retrain based on drift severity rather than just detection.

ONE CONCRETE EXAMPLE: Suppose you have a model with thirty features. You set the reference period to the last stable month of training data. For live traffic, you create daily chunks. For each numerical feature, the univariate drift calculator compares the chunk distribution against reference; for categorical features, it checks category shifts and unseen values. You also run a PCA reconstruction multivariate drift detector across all features. If two numerical features and one categorical feature cross their thresholds, the ranking module surfaces them by alert count. The on-call engineer inspects the top-ranked feature first and discovers a partner API started sending nulls as zeros, causing a mean shift that univariate detection caught immediately.