Self-attention and the Query, Key, Value matrices

WHAT THIS TESTS Whether you can explain attention precisely: what Q, K, and V each do and the exact sequence of operations that turns them into an output.

A GOOD ANSWER COVERS Every input token embedding is linearly projected three ways using learned matrices into a Query, a Key, and a Value. Intuitively the Query represents what this token is looking for, the Key represents what each token offers, and the Value is the content actually passed along. To compute attention for a position, take its Query and compute a dot product with the Key of every position; this yields a relevance score per pair. Divide each score by the square root of the key dimension to keep the dot products from growing large and pushing softmax into saturated regions with vanishing gradients. Apply softmax across the sequence so the scores become a probability distribution, the attention weights. The output for the position is the weighted sum of all Values using those weights. In matrix form this is softmax of Q times K transpose over root d, times V. Multi-head attention runs this in parallel across several subspaces and concatenates, letting different heads capture different relationships.

COMMON WRONG ANSWERS Saying attention just looks at other words without the Q-K-V mechanics. Mixing up which matrix scores and which carries content. Omitting the scaling factor or not explaining why it exists. Forgetting softmax is taken over the sequence dimension. Confusing self-attention with cross-attention.

LIKELY FOLLOW-UPS Why divide by root d? What does multi-head attention add? What is the time and memory complexity in sequence length? How does causal masking change the computation?

ONE CONCRETE EXAMPLE In the cat sat, the Query for sat dot-products with the Keys of the, cat, and sat. Suppose it scores highest against cat. After scaling and softmax, cat receives the largest weight, so the output for sat is dominated by cat's Value, encoding that the subject of sat is cat. Other heads might instead attend to positional or syntactic relations.