Three ways to cut LLM inference cost

WHAT THIS TESTS Whether you know real LLM serving optimizations at the model and infrastructure level, not just hardware swaps.

TECHNIQUE ONE QUANTIZATION Quantize model weights and activations from sixteen-bit to int8 or int4. This shrinks memory footprint, lets the model fit on smaller or fewer GPUs, and speeds memory-bound inference, usually with small accuracy loss when done with calibration. Lower precision directly lowers cost per token.

TECHNIQUE TWO CONTINUOUS BATCHING WITH PAGED ATTENTION LLM serving is dominated by GPU underutilization when requests are processed one at a time. Continuous, or in-flight, batching interleaves many concurrent requests at the token level so the GPU stays busy, and paged attention, popularized by vLLM, manages the KV cache in noncontiguous pages to eliminate memory fragmentation and fit more concurrent sequences. Together they sharply raise throughput per GPU, lowering cost per request.

TECHNIQUE THREE COMPRESSION OR ROUTING Distill the large model into a smaller student for the common case, or use a router that sends easy queries to a small cheap model and reserves the large model for hard ones. Speculative decoding, where a small draft model proposes tokens the big model verifies, similarly cuts cost while preserving quality. Caching repeated prompts and prefixes avoids recompute.

COMMON WRONG ANSWERS Only suggesting a cheaper GPU, generic horizontal autoscaling without raising per-GPU efficiency, or claiming quantization is free with no quality consideration.

LIKELY FOLLOW-UPS How much quality quantization costs, how paged attention reduces KV-cache waste, and when distillation is worth the effort.

ONE CONCRETE EXAMPLE Serving on vLLM with int4 weights and continuous batching lifts a single GPU from a handful of concurrent requests to dozens, and routing short factual queries to a distilled small model means the expensive large model only runs for genuinely hard prompts, cutting average cost per inference substantially.