Design auto drift detection and retraining

WHAT THIS TESTS This evaluates whether you can build a closed feedback loop that distinguishes data drift, concept drift, and infrastructure health, and wires detection to automated retraining.

A GOOD ANSWER COVERS Capture: log every request's features and the model's prediction, sampling if volume is huge. Baseline: store the training data's statistical profile. Detection: a scheduled job computes data-drift signals such as population stability index, KL divergence, or KS tests on feature distributions, and concept-drift signals such as accuracy, precision, or AUC decay once true labels arrive. Thresholds turn these into alerts. Action: a breach publishes an event that triggers a retraining pipeline, which pulls recent labeled data, retrains, evaluates against the incumbent, and canary-deploys only if it wins. Guardrails prevent retraining loops on noisy data.

COMMON WRONG ANSWERS Watching only CPU, latency, and 5xx rates. Assuming ground-truth labels are immediate. No baseline to compare against. Retraining automatically on every minor wobble, amplifying noise.

LIKELY FOLLOW-UPS How do you detect drift before labels arrive? How do you avoid alert fatigue? How do you prevent a feedback loop where the model influences future data? How do you roll back?

ONE CONCRETE EXAMPLE On AWS, SageMaker Model Monitor captures endpoint traffic to S3 and runs scheduled baselining jobs that emit drift metrics to CloudWatch. A CloudWatch alarm on a PSI breach triggers an EventBridge rule that starts a SageMaker Pipeline: it retrains, runs an evaluation step against the deployed model, and a conditional step registers and deploys the candidate behind a canary only when metrics improve.