Dynamic batching and the throughput-latency trade-off

WHAT THIS TESTS This evaluates whether you understand that GPUs are massively parallel, so batching trades individual request latency for aggregate throughput, and that time to first token is a distinct latency concern.

A GOOD ANSWER COVERS A GPU processes a batch of sequences in roughly the same time as a single one up to its compute limit, so running requests one at a time wastes most of the hardware. Dynamic batching has the server collect requests arriving within a short time window and group them into one batch sent to the model together, dramatically improving throughput, the number of requests served per second. The trade-off is latency. To form a larger batch the server may wait for more requests to arrive, and that queueing delay directly increases each request's latency and especially the time to first token, the gap between sending a prompt and seeing the first output token, which dominates perceived responsiveness in chat. Larger batches also take slightly longer per step. So there is a tension: bigger batches and longer wait windows maximize throughput and hardware efficiency but hurt latency; smaller batches and shorter windows keep latency low but waste GPU capacity. Continuous, or in-flight, batching improves on naive dynamic batching by adding new requests and evicting finished ones at every decoding step rather than waiting for the whole batch to finish, capturing most of the throughput gain with far less latency penalty.

COMMON WRONG ANSWERS Claiming larger batches reduce latency; conflating throughput with latency; ignoring the queueing wait to fill a batch; forgetting time to first token; not knowing continuous batching exists.

LIKELY FOLLOW-UPS Why does time to first token matter most in chat? How does continuous batching differ from static? How do you tune the batch window? How does the KV cache interact with batching?

ONE CONCRETE EXAMPLE Under light load the server processes each request almost immediately, low latency but poor GPU use. Under heavy load it batches dozens of requests: throughput soars, but a user whose request waited in the batching window sees a longer time to first token. Switching to continuous batching lets the server slot that user's request in at the next token step, keeping throughput high while cutting the first-token delay.