Inference Throughput: How Many Predictions Per Second?

WHY IT EXISTS A model that is fast for one user (low latency) might collapse under the load of thousands of simultaneous requests. Businesses need a way to measure and design for total system capacity, not just single-request speed. Inference throughput provides this metric, answering the question: 'How much work can this system handle over time?'.

THE MENTAL MODEL Think of a highway versus a single-lane road. The single-lane road might have a high speed limit of 70 mph (low latency), but it can only handle one car at a time in a given spot. A five-lane highway with the same speed limit has much higher throughput, moving far more cars per hour. Inference throughput is the highway's total capacity, while latency is the speed of one car.

HOW IT WORKS Throughput is measured in inferences per second (IPS) or requests per second (RPS). The most common technique to increase throughput is dynamic batching. Instead of processing each incoming request individually, the system groups multiple requests into a single batch and feeds them to the model in one forward pass. This is far more efficient for hardware like GPUs, which are optimized for parallel computation, dramatically increasing the number of inferences completed per second.

WHEN TO USE IT Prioritize throughput when the total volume of predictions is the key business metric and individual latency is less critical. This is common in offline batch processing, like analyzing a day's worth of financial transactions for fraud. It's also vital for near-real-time systems with massive request volumes, such as serving recommendations on an e-commerce homepage or moderating content on a social media feed.

WHEN NOT TO USE IT Do not optimize for throughput at the expense of latency in applications where immediate response is non-negotiable. For example, in an autonomous vehicle, the system deciding whether to brake cannot wait for a batch of sensor readings to fill up. Similarly, an interactive chatbot feels slow and unresponsive if its replies are delayed by batching.

ONE CANONICAL EXAMPLE A video streaming service uses a model to recommend what a user should watch next. The service has millions of active users. The system's success depends on its ability to generate recommendations for everyone concurrently (high throughput). By batching requests from many users, the service can efficiently utilize its GPU cluster to serve millions of recommendations per minute, even if any single user's recommendation takes a few hundred milliseconds longer than it would have in a non-batched system.