What is the KV cache and why does it matter for serving LLMs?

### What this tests This question probes your understanding of the practical performance characteristics of the Transformer architecture, not just its theoretical components. It specifically tests if you know how autoregressive generation works under the hood and the critical optimization that makes real-time LLM applications feasible by avoiding redundant computation in the self-attention layers.

### A good answer covers * **The Role of K and V in Attention:** Start by explaining that in each Transformer layer, the self-attention mechanism computes three matrices from the input token embeddings: Query (Q), Key (K), and Value (V). The attention scores are computed using Q and K, and these scores are then used to create a weighted sum of the V vectors. * **The Redundancy in Autoregressive Generation:** During inference, LLMs generate tokens one by one. To generate token `N+1`, the model processes all tokens from `1` to `N`. The K and V vectors for all these `N` tokens are calculated. To generate token `N+2`, the model would naively re-process tokens `1` to `N+1`, wastefully re-calculating the K and V vectors for the first `N` tokens. * **The KV Cache Solution:** The KV cache is a simple but powerful optimization. It's a memory buffer (typically in VRAM) that stores the Key and Value tensors for all tokens in the sequence generated so far. At each subsequent step, the model only needs to compute the K and V vectors for the *single new token* and append them to the cache. The attention calculation then uses the full cached K and V tensors. * **The Performance Impact:** This optimization changes the complexity of each generation step. Without the cache, processing a sequence of length `N` is roughly `O(N^2)`. With the cache, generating the next token only requires attending to the `N` cached tokens, making the step `O(N)`. This dramatically reduces latency and the total computation required for generating a full response.

### Common wrong answers * **Vague Descriptions:** Describing it as just "caching previous computations" without specifically naming the Key and Value tensors from the attention mechanism. * **Confusing Cache Types:** Mixing it up with application-level prompt caching or memory caching of model weights. The KV cache is specific to the state of a single generation sequence within the GPU. * **Ignoring the 'Why':** Explaining what it is but failing to connect it to the autoregressive generation process, which is the entire reason the cache is necessary.

### Follow-up the interviewer might ask * "What is the memory footprint of the KV cache, and how does that create a trade-off with context length?" * "How do optimizations like Grouped-Query Attention (GQA) or Multi-Query Attention (MQA) affect the KV cache?"

### One concrete example For a Llama-2 7B model (32 layers, 32 heads, head dimension 128) generating a sequence with a 4096-token context window:

* **Cache Size per Layer:** `2 (K and V) * 4096 (tokens) * 32 (heads) * 128 (head dim) * 2 (bytes/fp16) = ~67 MB` * **Total Cache Size:** `67 MB/layer * 32 layers = ~2.1 GB`

This shows that for just a single user with a 4k context, the KV cache consumes over 2 GB of VRAM, making it a primary factor in determining how many concurrent users a single GPU can serve.