Expose a trained model as a simple web service

What's really being asked

This question probes whether you can bridge the gap between a trained model artifact and a production-grade service. Interviewers want to see that you think in terms of the MLOps lifecycle, specifically the packaging, deployment, and integration stages, rather than treating serving as an afterthought. They are looking for awareness of scalability, observability, and dependency isolation.

The full answer

A strong response walks through five layers in order. First, serialization: export the trained model into a stable format such as ONNX, TorchScript, or TensorFlow SavedModel so the artifact is decoupled from the training notebook. Second, packaging: bundle the serialized artifact, inference code, and dependency manifest into a Docker image to create a portable model server. Third, serving infrastructure: expose REST endpoints through an API gateway that delegates traffic to a load balancer, which then distributes requests across multiple worker processes or nodes. Fourth, the runtime: each worker loads the model into memory and performs inference, ideally using an ML-optimized runtime rather than a generic web framework for large models. Fifth, observability: add a monitoring endpoint that tracks request latency, throughput, input data distribution, and prediction drift. The candidate should also mention that this architecture isolates heavy ML dependencies from the main application stack and enables independent versioning and rollback.

The mistakes people make

The biggest red flag is stopping at a raw Flask or FastAPI script with model.load and app.run, with no mention of containers, serialization, or horizontal scaling. Another weak pattern is conflating training dependencies with serving dependencies, suggesting the server image includes Jupyter, full GPU training stacks, or data pipelines. Failing to mention health checks, request validation, or a monitoring endpoint also signals a lack of production experience. Finally, suggesting a model server for pure batch processing or a trivial single-instance workflow shows poor architectural judgment.

What usually comes next

Expect the interviewer to ask how you would handle a model that is too large for a single container, how to implement A-B testing between two model versions, or how to protect the endpoint from traffic spikes. They may also dig into cold-start latency, GPU utilization strategies, or how to update the model without dropping traffic.

A concrete example

Suppose you have a PyTorch computer vision model. You would trace it to TorchScript, then copy the artifact into a Docker image built on an ML-optimized server like TorchServe or Triton. The container exposes a REST endpoint on port 8080 behind an Nginx API gateway. A load balancer spreads incoming requests across three worker replicas. Each worker keeps the model resident in memory and returns predictions in under 100 milliseconds. A monitoring sidecar scrapes latency and logs input image distributions to detect data drift.