Design cost-effective inference for spiky traffic without idle GPUs

WHAT THIS TESTS: Your ability to design ML inference infrastructure that avoids paying for idle GPU capacity during traffic troughs while still delivering low latency during unpredictable spikes. The interviewer wants to see that you treat deployment updates as capacity management events and that you know how to use SageMaker native guardrails to minimize risk and waste.

A GOOD ANSWER COVERS four things in order. First, component choice: anchor on SageMaker real-time endpoints with production variants, which the AWS best practices guide identifies as the standard mechanism for real-time single-model serving and for distributing traffic across model versions. Second, capacity and update strategy: use blue/green deployment guardrails as the default pattern. You stand up a green fleet, shift endpoint traffic from the blue fleet to the green fleet, monitor Amazon CloudWatch alarms during the baking period, and then have SageMaker delete the blue fleet once the shift succeeds. This eliminates the idle capacity of the old fleet rather than leaving it running. Third, validation under load: before shifting critical traffic, use shadow tests to mirror production requests to the new model without serving responses, or use A/B tests when you have closed-loop business metrics. This ensures you do not commit expensive GPU capacity to an unproven model. Fourth, operational safety: configure CloudWatch alarms carefully during the baking period so that latency spikes or errors trigger an automatic rollback, preserving availability exactly when traffic is peaking.

COMMON WRONG ANSWERS include proposing always-on GPU clusters with naive horizontal auto-scaling, which leaves costly idle capacity during troughs; recommending in-place model updates that create downtime during spikes; or ignoring the baking period and alarms that the reference highlights as essential for safe traffic shifts. Another red flag is suggesting serverless or multi-model SageMaker capabilities not discussed in the provided best practices.

LIKELY FOLLOW-UPS are how you would decide between all-at-once versus gradual traffic shifting for a latency-sensitive GPU workload; what CloudWatch metrics you would alarm on during the baking period beyond simple error rate; and how shadow testing affects inference cost if you must run two fleets in parallel temporarily.

ONE CONCRETE EXAMPLE is a retail recommendation endpoint that sees ten-minute traffic spikes after marketing pushes. When the team deploys a new model version, they use SageMaker blue/green guardrails from the AWS guide. They launch a green fleet, configure CloudWatch alarms for latency and errors, and execute an all-at-once traffic shift after a ten-minute baking period passes. Once traffic is fully on green, SageMaker deletes the blue fleet, ensuring no idle GPU capacity remains from the old version. During the spike itself, the endpoint uses its production variant configuration to distribute load across the active fleet while the deployment guardrails ensure model updates do not add extra idle capacity during the peak.