Explain Q, K, and V matrices in self-attention

WHAT THIS TESTS: This question probes whether you see self-attention as a differentiable information-retrieval operation rather than a rote matrix formula. The interviewer wants to know if you understand why three separate projections are necessary and how the Q-K-V abstraction enables each token to pull contextual information from the entire sequence.

A GOOD ANSWER COVERS: First, derivation: starting from input embeddings of size 512 in the original Transformer, Q, K, and V are produced by multiplying the same input embedding matrix by three separate learned weight matrices. Second, intuition for names: Query is the vector representing the current token's information need, Key is a searchable index that describes what each token offers, and Value is the actual content or representation to be retrieved. Third, interaction: compute a compatibility score between a Query and every Key via dot product, scale and softmax to get attention weights, then form a weighted sum of Value vectors. Fourth, why this works: the learned projections let the model decide which aspects of a token to use for matching versus which aspects to pass forward as content.

COMMON WRONG ANSWERS: A major red flag is saying Q, K, and V come from different inputs or previous layers rather than being three views of the same source. Another is claiming the projections are hand-designed or fixed instead of trainable linear maps. Some candidates describe the dot product as comparing raw embeddings and forget to mention the learned weight matrices entirely. Confusing self-attention with cross-attention by suggesting Q comes from the decoder while K and V come from the encoder is also a mistake in this context.

LIKELY FOLLOW-UPS: The interviewer may ask why we need three projections instead of one or two, which opens a discussion about representational capacity and the need to decouple matching from content. They might ask about the scaling factor and why it matters for softmax saturation. Another follow-up is multi-head attention: how splitting into multiple heads changes the Q-K-V shapes and what parallel attention patterns buy you. You might also be asked to contrast this with RNN hidden-state aggregation.

ONE CONCRETE EXAMPLE: Imagine encoding the word bank in the sentence I sat by the river bank. The input embedding for bank is multiplied by a learned weight matrix to produce a Query asking for nature-related context. Every token in the sentence produces a Key via its own projection; the Keys for river and water would have high dot-product similarity with bank's Query, while the Key for sat would score low. Those scores become weights via softmax, and the output for bank becomes a weighted sum of all Value vectors, heavily favoring river's Value. The result is a context-aware representation of bank that differs from its representation in I went to the bank to deposit cash.