How the KV cache speeds transformer generation

WHAT THIS TESTS The interviewer checks whether you understand the mechanics of autoregressive generation and the dominant inference optimization in transformer serving. It distinguishes those who know how attention is actually computed step by step.

A GOOD ANSWER COVERS In a decoder, generating each new token requires self-attention over all previous tokens. For the current token's query, the model needs the key and value vectors of every prior token. Those keys and values depend only on the earlier tokens, which are already fixed once generated, so they do not change. Without caching, at each step the model would re-encode the entire prefix and recompute every key and value, doing redundant work that grows quadratically with sequence length. The KV cache stores the key and value tensors for all past tokens, per layer and per head. At each new step the model computes only the new token's query, key, and value, appends the new key and value to the cache, and attends over the cached set. This turns per-step work from recomputing the whole prefix into a single token's projection plus an attention read, reducing per-token cost from quadratic to linear in the current length and dramatically cutting latency. The trade-off is memory: the cache grows with sequence length, batch size, layers, and heads, and can dominate GPU memory for long contexts.

COMMON WRONG ANSWERS Saying it caches the model's text outputs or logits rather than key and value tensors. Claiming it speeds up training, it applies to incremental decoding. Forgetting the memory cost it introduces.

LIKELY FOLLOW-UPS How large does the cache get, proportional to layers times heads times head-dim times tokens times batch. Why is this the motivation for PagedAttention, fragmentation of cache memory. How do GQA and MQA shrink it, fewer KV heads.

ONE CONCRETE EXAMPLE Generating the 1000th token without a cache would recompute keys and values for the prior 999 tokens every step. With the KV cache those are stored once, so step 1000 computes a single token's projections and reads the cache, slashing redundant compute.