Multi-Head Attention: Seeing Data From Multiple Angles

Why it exists

A single attention mechanism can struggle to capture the many overlapping relationships in a sequence. For example, a sentence has grammatical structure, semantic meaning, and long-range dependencies all at once. A model needs a way to process these different layers of meaning simultaneously.

The mental model

Imagine you're reading a complex legal document. You might read it once to identify the parties involved, a second time to understand the core obligations, and a third time to note the deadlines. Multi-head attention does this in parallel. It creates several "heads," where each head is its own attention mechanism that learns to focus on a different aspect or relationship within the input data. The model then combines these different perspectives for a richer understanding.

How it works

Instead of calculating one set of attention weights for an input sequence, multi-head attention first splits the model's representational space into several subspaces. For each subspace, it initializes a separate attention "head". Each head independently calculates attention scores for the sequence, learning to focus on a specific type of relationship. The outputs from all heads are then concatenated and passed through a final linear layer to produce a unified output that incorporates insights from all perspectives.

When to use it

Use it when your data contains complex, multi-layered relationships. It is the core component of the Transformer architecture, making it the default for state-of-the-art NLP tasks like translation, summarization, and question answering. It is also foundational to Vision Transformers (ViTs) for finding relationships between different patches of an image.

When not to use it

For very simple sequences or when computational cost is a primary constraint. The parallel computations add significant overhead compared to a single attention mechanism. For simpler problems or small datasets where it might overfit, a more basic model like an RNN or a single-head attention mechanism could be more efficient and just as effective.

One canonical example

In the sentence "The animal didn't cross the street because it was too tired," one attention head might learn that "it" refers to "the animal." Simultaneously, another head might learn the causal link between "didn't cross" and "tired." Multi-head attention allows the model to understand both the grammatical reference and the semantic reason in parallel, which a single attention mechanism might miss or conflate.