How would you design a system to detect training-serving skew using model registry metadata?

WHAT THIS TESTS: The interviewer wants to see if you can bridge offline training and online serving environments at scale. Training-serving skew is not just generic data drift; it specifically refers to differences in how features are computed, encoded, or filtered between the training pipeline and the inference path. The question tests your ability to design a feedback loop that uses the model registry as a source of truth for baseline statistics and continuously validates production traffic against those baselines.

A GOOD ANSWER COVERS: First, the metadata schema in the model registry should store immutable per-feature statistics from training such as mean, standard deviation, categorical cardinality, quantiles, and expected value ranges, versioned alongside the model artifact. Second, the system needs a logging layer that captures the exact feature vectors seen at inference time, not just predictions, because skew happens in the input space; this usually means async logging to a feature store or object storage to avoid latency impact. Third, a compute layer runs periodic or streaming jobs that align production feature distributions with training metadata using statistical tests like Population Stability Index for numerical shifts, Chi-squared for categorical shifts, or Kolmogorov-Smirnov for distribution shape changes, with configurable thresholds per feature. Fourth, an action layer translates drift severity into tiered responses such as alerts for minor drift, automatic promotion of a challenger model for moderate drift, and full rollback or circuit breaking for severe skew.

COMMON WRONG ANSWERS: Proposing schema validation as the primary defense is a red flag because matching types and column names does not catch distributional shifts. Another weak pattern is suggesting manual ad-hoc comparisons or SQL spot checks instead of automated continuous monitoring. Some candidates also forget that inference logging must be asynchronous so that skew detection does not add P99 latency to the serving path.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle high-cardinality categorical features where standard binning fails, how you distinguish training-serving skew from natural concept drift, or how you would reduce false positives when seasonality causes expected distribution shifts. They might also probe whether you would block inference requests synchronously or surface drift asynchronously.

ONE CONCRETE EXAMPLE: Imagine a fraud model trained on transaction amounts with a registry-stored mean of 120 and a standard deviation of 45. In production, a partner integration starts sending bulk corporate transactions, pushing the seven-day rolling mean to 890. The monitoring job compares the production distribution against the registry baseline using PSI and detects a score of 0.35, well above the 0.25 threshold. The system triggers a shadow fallback to the previous model version and pages the on-call to investigate the feature engineering pipeline for the transaction amount variable.