CPU versus GPU serving: cost, latency, throughput

What's really being asked

This checks whether you can pick inference hardware based on workload economics rather than a reflex that GPUs are always better. It rewards understanding why GPUs help and when they are wasted.

The full answer

GPUs are built for massively parallel operations, so for large deep-learning models with heavy matrix multiplication they deliver far higher throughput than CPUs. That advantage is largest when you batch many requests together, because the GPU processes the batch in parallel and amortizes its overhead. The downsides are high cost per hour and poor utilization if traffic is low or requests arrive one at a time, since a GPU serving a single small input may sit mostly idle while you still pay for it. CPUs are cheaper, available everywhere, and well suited to small or classical models, low request volume, or strict single-request latency where batching is impossible. The right choice depends on model size and architecture, request volume and burstiness, whether you can batch, and your latency budget against your cost ceiling. Mention that GPU latency for a single tiny request can even exceed CPU latency due to data-transfer overhead.

The mistakes people make

Saying GPU is always faster and therefore always correct. Ignoring batching, which is what makes GPUs cost-effective. Forgetting that an underutilized GPU is expensive and that data transfer to the GPU adds latency for small payloads.

What usually comes next

How does dynamic batching change the calculus? When would you autoscale GPUs versus CPUs? What about quantizing the model to run on CPU?

A concrete example

A large image-classification model serving thousands of requests per second benefits from a GPU with dynamic batching, achieving high throughput at acceptable per-image cost. The same model behind an internal tool handling ten requests an hour is cheaper on a CPU, because a GPU would sit idle ninety-nine percent of the time while still billing every hour.