Flask/Gunicorn vs Triton/TorchServe for serving

WHAT THIS TESTS Whether you can weigh control and simplicity against throughput and operational features when selecting how to serve a model, rather than reflexively picking one tool.

A GOOD ANSWER COVERS A general-purpose stack like Flask behind Gunicorn wraps the model in a normal HTTP handler. Its strengths are simplicity, full control over request and response logic, easy integration of arbitrary Python preprocessing or business rules, and minimal new dependencies. Its weaknesses are that you must build the hard parts yourself: efficient request batching, concurrent execution across GPUs, model versioning, health and readiness, and rich inference metrics. Gunicorn's process model also wastes GPU memory if each worker loads its own model copy. Dedicated serving frameworks such as NVIDIA Triton and TorchServe are purpose-built for inference. They provide dynamic batching that groups concurrent requests to maximize GPU utilization, concurrent model instances, multi-framework support, model versioning and hot reload, and standardized metrics endpoints. Triton additionally supports model ensembles and many backends.

COMMON WRONG ANSWERS Always choosing the heavyweight framework even for a tiny, low-traffic, CPU-only model adds operational complexity for no benefit. Conversely, scaling a hand-rolled Flask service to high GPU throughput and reimplementing dynamic batching badly is a frequent failure.

LIKELY FOLLOW-UPS How does dynamic batching trade latency for throughput? How do you serve multiple models or versions behind one endpoint? When would you put a thin Flask layer in front of Triton for custom pre and post processing?

ONE CONCRETE EXAMPLE A small internal classifier handling a few requests per minute on CPU ships fastest as a Flask plus Gunicorn endpoint. A customer-facing vision model needing thousands of GPU inferences per second, several model versions, and high utilization is far better on Triton, where dynamic batching and concurrent instances keep the GPU saturated and versioning is built in.