Why Multi-Head Attention
rationale for splitting attention into heads.
multiple heads attend to different subspaces and relations in parallel, which one big head averages away.
claiming more heads is always better or that it raises total compute.
WHAT THIS TESTS This checks deeper understanding of attention design choices, specifically why the Transformer uses many small heads instead of one large one.
A GOOD ANSWER COVERS A single attention head computes one set of softmax weights, which tends to collapse toward a single, averaged notion of relevance; it can attend strongly to one kind of relationship at a time. Multi-head attention projects the queries, keys, and values into several lower-dimensional subspaces using separate learned matrices, runs attention independently in each head, then concatenates the results and applies a final linear projection. This lets different heads specialize: one might track syntactic dependencies, another positional proximity, another coreference. Crucially the model dimension is split across heads, so each head is narrower and the total computation is comparable to one full-size head, you get diversity of attention patterns essentially for free.
COMMON WRONG ANSWERS Claiming multi-head attention multiplies the cost by the number of heads; the dimension is partitioned, so it does not. Asserting more heads is always better; beyond a point heads become redundant and some can be pruned with little loss. Saying each head sees the full model dimension rather than a subspace.
LIKELY FOLLOW-UPS How is the head dimension typically chosen relative to model dimension and head count? What does research on head redundancy and pruning show? How do the concatenation and output projection recombine heads? How does this relate to grouped-query or multi-query attention used to speed inference?
ONE CONCRETE EXAMPLE With a model dimension of 512 and 8 heads, each head works in a 64-dimensional subspace. Processing a sentence, one head might place most of its weight on the immediately preceding word capturing local syntax, while another links a verb to its distant subject. A single 512-wide head would have to compromise all these patterns into one averaged weighting, losing the specialization, whereas the eight heads capture them simultaneously at the same total cost.
Interview question
Why does multi-head attention generally not cost much more than a single full-width attention head?
- a.Softmax is skipped in all but one head to save computation
- b.Extra heads share a single query-key-value projection matrix
- c.The model dimension is split across heads, so each head operates in a smaller subspaceCorrect
- d.Heads are computed sequentially, reusing the same weights each time
Why? this is the answer
The total dimension is partitioned among heads, keeping aggregate compute comparable to one full head while gaining diverse attention patterns. Heads run in parallel with their own projections, not shared weights, and each still uses softmax.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #multi-head-attention
- #transformers
- #attention
- #self-attention
- #nlp
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles