Deploying a real-time inference endpoint

WHAT THIS TESTS This assesses whether you understand the full set of components needed to serve a model reliably in production, beyond simply having trained weights.

A GOOD ANSWER COVERS Start by packaging the deployable unit: the trained model artifact plus an inference script that handles loading, input preprocessing, prediction, and output formatting, bundled into a container image, often using the service's prebuilt serving containers for common frameworks. Register the model and create an online or real-time endpoint. Configure the compute: choose CPU or GPU instances sized to the model's latency and memory needs, set the instance count, and enable autoscaling so capacity tracks request volume rather than being fixed. Define the request and response contract and any batching. Set health and readiness probes so unhealthy instances are replaced, and secure the endpoint with authentication and network controls. For deployment safety, do not flip all traffic at once: use a canary or blue-green strategy that shifts a small percentage to the new version, validates metrics, then ramps up, with fast rollback. Finally attach observability: log requests and predictions, track latency, throughput, and error rates, and monitor for data and prediction drift that signals the model is degrading and needs retraining.

COMMON WRONG ANSWERS Deploying a fixed single instance with no autoscaling, which either wastes money or falls over under load. Skipping a canary or blue-green rollout and risking a bad version on all traffic. Omitting monitoring and drift detection. Forgetting that preprocessing must match training exactly, or the endpoint returns subtly wrong results.

LIKELY FOLLOW-UPS Real-time versus batch versus async endpoints. How do you keep preprocessing consistent with training. How do you autoscale to zero for sporadic traffic.

ONE CONCRETE EXAMPLE You containerize a model with its inference handler, deploy it to a SageMaker or Azure ML online endpoint on a GPU instance with autoscaling between two and ten instances, route ten percent of traffic to the new version as a canary while watching latency and error rate, then promote to one hundred percent and keep dashboards plus drift alerts running.