Design a system to monitor a real-time prediction service for feature drift

What's really being asked

This question evaluates whether you understand that production ML systems fail silently when input distributions shift. Interviewers want to see you can separate inference from observability, choose appropriate statistical metrics, and design an alert mechanism that minimizes false positives without delaying detection. The core signal is architectural maturity: logging must not block prediction latency, comparisons must happen continuously rather than in ad-hoc notebooks, and thresholds must adapt to seasonality.

The full answer

First, the instrumentation layer. The prediction service should emit a structured event per request containing the feature vector, prediction timestamp, model version, and optional prediction confidence. These events flow asynchronously through a message bus like Kafka or Kinesis so the inference path stays under single-digit millisecond latency. Second, the reference store. You need an immutable profile of the training distribution, stored as histograms, quantiles, or parametric summaries per feature, versioned by model training run. Third, the drift computation engine. A separate consumer reads the stream in sliding windows, say five minutes or ten thousand requests, and computes univariate tests such as Population Stability Index, Kolmogorov-Smirnov, or Wasserstein distance for numerical features, and chi-squared for categorical features. Multivariate drift via autoencoder reconstruction error or maximum mean discrepancy is a bonus. Fourth, alerting logic. Use dynamic thresholds based on rolling z-scores or percentile bands rather than hard cutoffs, and route alerts through an on-call rotation with severity levels. A page triggers only after sustained drift across multiple features or windows, while a warning logs to a dashboard for review.

The mistakes people make

A red flag is proposing synchronous drift computation inside the prediction API, which adds unacceptable latency at scale. Another mistake is waiting for ground-truth labels before acting; feature drift is an input-level problem and must be caught before label delay, which can be days or weeks. Candidates also err by suggesting a single global threshold for all features, ignoring that sparse categorical features and dense numerical features behave differently. Finally, storing raw feature logs indefinitely in the hot path without sampling or TTL is an operational and cost anti-pattern.

What usually comes next

How do you handle high-cardinality categorical features? What is your sampling strategy if traffic is millions of QPS? How would you distinguish feature drift from legitimate seasonality? When would you trigger an automatic rollback versus just an alert? How do you monitor embeddings or unstructured inputs like text or images?

A concrete example

Imagine a fraud model using transaction amount and merchant category. The training reference shows transaction amounts are log-normal with a mean of 4.5 and standard deviation of 1.2. At deployment, the service logs every request to a Kafka topic. Every five minutes, a Flink job computes the KS statistic between the reference histogram and the current window. If the KS statistic exceeds 0.15 for two consecutive windows, a P2 alert fires in PagerDuty. The on-call engineer sees a dashboard showing the distribution shifted bimodal because a new merchant integration launched, prompting a feature engineering review rather than an immediate retrain.