Explain dynamic batching in inference servers and its trade-off

WHAT THIS TESTS: This question tests whether you understand the scheduling layer of an inference server and can reason about the throughput-versus-latency trade-offs that define production GPU serving. Interviewers want to see that you know dynamic batching is not magic; it is a heuristic scheduling policy that balances utilization against user-visible latency. They also want to know if you can distinguish it from continuous batching, which is the state-of-the-art approach in modern LLM serving frameworks.

A GOOD ANSWER COVERS: First, define dynamic batching as a request-collection strategy that waits for one of two conditions: either a maximum batch size is reached or a timeout window expires. Whichever comes first triggers execution. Second, explain the performance benefit: by amortizing model weight loading across multiple requests, it raises throughput compared to single-request or static batching. Third, state the primary trade-off clearly: because all sequences in a batch start and finish together, short generations are held hostage by the longest one in that batch. This means tail latency can spike and GPU resources may sit idle while waiting for the straggler. Fourth, contrast it with continuous or in-flight batching, where finished sequences are evicted and replaced immediately without waiting for the entire batch to complete.

COMMON WRONG ANSWERS: A major red flag is conflating dynamic batching with continuous batching. Another is claiming that dynamic batching eliminates tail latency or solves head-of-line blocking; it does not, because the entire batch is gated by the slowest request. Some candidates also describe it as purely size-based without mentioning the timeout window, which misses the core mechanism that prevents indefinite stalling. Finally, saying it improves latency is usually wrong; it improves throughput at the cost of latency.

LIKELY FOLLOW-UPS: The interviewer may ask how you would tune the timeout and max batch size for a specific workload, or what metrics you would monitor to verify the setting. They might also ask when you would prefer dynamic batching over continuous batching, or how padding and variable sequence lengths affect GPU efficiency in a dynamic batch. Another common pivot is to ask about memory pressure: larger batches increase KV cache consumption, so how do you prevent out-of-memory errors?

ONE CONCRETE EXAMPLE: Imagine an inference server with a max batch size of 8 and a timeout of 10 milliseconds. At time zero, one request arrives. Under static batching, the GPU sits idle until all 8 slots fill, which could take hundreds of milliseconds. Under dynamic batching, if only 3 more requests arrive within 10 milliseconds, the server launches a batch of 4 at the timeout. Throughput rises because the GPU processes four requests in one forward pass instead of four separate passes. However, if those four requests generate 10, 50, 100, and 200 tokens respectively, the first three finish their actual generation early but cannot return results until the 200-token request completes. That idle waiting time for the short requests is the trade-off.

Source: bentoml.com