Three techniques to cut LLM inference latency

WHAT THIS TESTS This checks whether you know concrete, software-level inference optimizations beyond buying faster GPUs, and whether you can articulate each mechanism and its accuracy cost.

A GOOD ANSWER COVERS First, quantization: store and compute weights, and sometimes activations, in lower precision such as INT8 or 4-bit instead of 16-bit. This shrinks memory bandwidth and speeds matrix multiplies, the dominant cost. With good calibration accuracy loss is small, but aggressive quantization or unhandled outlier activations can degrade quality. Second, batching and KV caching: the key-value cache stores past attention states so each new token does not recompute the whole sequence; continuous, or in-flight, batching dynamically packs many concurrent requests so the GPU is never idle waiting on stragglers, sharply raising throughput and lowering average latency without touching accuracy. Third, speculative decoding: a small, cheap draft model proposes several tokens ahead, and the large target model verifies them in a single parallel forward pass, accepting the matching prefix. Because the target model still validates every token, output quality is unchanged; the speedup comes from generating multiple tokens per expensive step. Other valid options include FlashAttention and tensor parallelism.

COMMON WRONG ANSWERS Only proposing bigger or more GPUs; vaguely saying use a smaller model without specifics; claiming quantization never affects accuracy; confusing speculative decoding with sampling temperature; ignoring that continuous batching is about scheduling.

LIKELY FOLLOW-UPS Why does speculative decoding preserve outputs? How do outlier activations break INT8? How does continuous batching differ from static batching? Which technique helps time to first token versus throughput?

ONE CONCRETE EXAMPLE A chat service quantizes its model to INT8 for a memory and speed win, enables continuous batching so dozens of users share each GPU efficiently, and adds speculative decoding with a tiny draft model. End-to-end latency drops markedly while answer quality stays effectively unchanged, all without new hardware.