Why Multi-Head Attention

WHAT THIS TESTS This checks deeper understanding of attention design choices, specifically why the Transformer uses many small heads instead of one large one.

A GOOD ANSWER COVERS A single attention head computes one set of softmax weights, which tends to collapse toward a single, averaged notion of relevance; it can attend strongly to one kind of relationship at a time. Multi-head attention projects the queries, keys, and values into several lower-dimensional subspaces using separate learned matrices, runs attention independently in each head, then concatenates the results and applies a final linear projection. This lets different heads specialize: one might track syntactic dependencies, another positional proximity, another coreference. Crucially the model dimension is split across heads, so each head is narrower and the total computation is comparable to one full-size head, you get diversity of attention patterns essentially for free.

COMMON WRONG ANSWERS Claiming multi-head attention multiplies the cost by the number of heads; the dimension is partitioned, so it does not. Asserting more heads is always better; beyond a point heads become redundant and some can be pruned with little loss. Saying each head sees the full model dimension rather than a subspace.

LIKELY FOLLOW-UPS How is the head dimension typically chosen relative to model dimension and head count? What does research on head redundancy and pruning show? How do the concatenation and output projection recombine heads? How does this relate to grouped-query or multi-query attention used to speed inference?

ONE CONCRETE EXAMPLE With a model dimension of 512 and 8 heads, each head works in a 64-dimensional subspace. Processing a sentence, one head might place most of its weight on the immediately preceding word capturing local syntax, while another links a verb to its distant subject. A single 512-wide head would have to compromise all these patterns into one averaged weighting, losing the specialization, whereas the eight heads capture them simultaneously at the same total cost.