Why Transformers use multi-head attention

WHAT THIS TESTS This checks whether you understand attention beyond the formula and can explain why parallel heads improve representational capacity in Transformers.

A GOOD ANSWER COVERS A single attention mechanism computes one set of attention weights and produces a single weighted average of the value vectors. That forces all relationships between tokens through one representation subspace, so competing patterns, such as agreement between a verb and its subject versus a pronoun and its antecedent, must be averaged together and blur. Multi-head attention instead linearly projects the queries, keys, and values into several distinct, lower-dimensional subspaces, runs scaled dot-product attention independently in each, and concatenates the results before a final projection. Conceptually each head can specialize: some heads attend to adjacent tokens capturing local syntax, others link distant tokens for coreference or long-range dependencies, others track positional or semantic relations. Because the per-head dimension is the model dimension divided by the number of heads, the total cost is comparable to one full-size head while gaining diversity.

COMMON WRONG ANSWERS Saying multi-head attention is just more parameters for raw capacity. Claiming every head learns the same thing, defeating the purpose. Confusing the number of heads with the number of layers. Believing heads operate on the full dimension rather than projected subspaces.

LIKELY FOLLOW-UPS Why divide the model dimension across heads. What evidence shows heads specialize. Are all heads necessary, given pruning research. How does this relate to the scaling factor in scaled dot-product attention.

ONE CONCRETE EXAMPLE In the sentence the animal did not cross the street because it was too tired, resolving what it refers to needs a head that connects it back to animal. Meanwhile another head in the same layer might attend each word to its immediate neighbors to capture local phrasing. A single head would have to compromise between these, whereas eight heads let the model represent both relationships simultaneously and combine them.