How would you implement shadow deployment and which metrics justify promotion?

WHAT THIS TESTS: Whether you understand how to validate a machine learning model in production without exposing users to risk, and whether you know the difference between operational metrics and downstream business metrics. Shadow deployment is specifically for situations where there is no closed-loop feedback tying a single prediction to a business outcome, so the interviewer is looking for architectural discipline and metric selection.

A GOOD ANSWER COVERS: Four things in order. First, the traffic mirroring mechanism: duplicate incoming requests to a shadow variant running alongside the production model, serve only the production response to the caller, and capture the shadow predictions for offline evaluation. Second, operational guardrails: monitor p99 latency, error rate, throughput, and resource utilization such as CPU, GPU, and memory to confirm the new model does not degrade infrastructure performance or starve the production variant. Third, model quality evaluation: compare prediction distributions for drift, measure accuracy or RMSE against delayed ground truth labels, and check for calibration decay. Fourth, promotion criteria: define a bake period, for example 24 to 48 hours, require zero critical alarms, and use blue-green traffic shifting to move 100 percent of traffic only after the shadow passes all thresholds.

COMMON WRONG ANSWERS: Saying you will serve shadow traffic to a fraction of users, which is actually canary or A/B testing. Claiming you can measure live revenue, conversion, or click-through rate directly from a shadow model, which is impossible because the predictions are never served. Ignoring the cost and latency impact of running double inference on every request. Failing to explain how ground truth labels will be joined to shadow predictions later.

LIKELY FOLLOW-UPS: How do you join ground truth labels to shadow predictions when feedback arrives hours or days later? At what point does shadow testing become too expensive and you should switch to A/B testing? How do you prevent the shadow variant from increasing production latency if inference is synchronous? How would you handle models with side effects or stateful behavior?

ONE CONCRETE EXAMPLE: Using Amazon SageMaker, you configure an endpoint with two production variants. The existing model receives 100 percent of traffic and returns responses to the application. The new model is added as a shadow variant that receives mirrored requests but returns no response to the caller. Shadow outputs are logged to Amazon S3. You set CloudWatch alarms on p99 latency staying under 100 milliseconds, error rate below 0.1 percent, and GPU utilization under 80 percent. After 48 hours with no alarms and an offline evaluation showing no prediction drift, you execute a blue-green deployment to shift all traffic to the new model.

Source: aws.amazon.com