Cross-Attention: How Models Connect Two Ideas
Cross-attention lets a model, like a translator, focus on relevant parts of an input (e.g., a sentence) to generate an output (the translation). It's used in machine translation and image captioning. The footgun is confusing it with self-attention.
WHY IT EXISTS: Early encoder-decoder models compressed an entire input sequence, like a sentence, into a single, fixed-size vector. This created an information bottleneck, especially for long inputs. Cross-attention was introduced to allow the decoder to access information from the entire input sequence at every step, not just from one compressed summary.
THE MENTAL MODEL: Imagine a human translator working on a long sentence. They don't just read it once and then write the translation from memory. As they write each translated word, their eyes dart back to the source text, focusing on the most relevant words for what they're about to write. Cross-attention is the computational version of this "looking back" process. The decoder queries the encoder's output to find what's important.
HOW IT WORKS: Cross-attention operates on three sets of vectors: Queries (Q), Keys (K), and Values (V). In an encoder-decoder setup, the decoder generates a Query vector representing what it needs to produce the next output token. The encoder provides the Key and Value vectors for every token in the input sequence. The model calculates a score between the decoder's Query and every encoder Key. These scores are then used to create a weighted sum of the encoder's Value vectors, producing a context vector that is highly relevant to the current decoding step.
WHEN TO USE IT: Use it for tasks that map one representation to another, especially if they have different lengths. This is common in sequence-to-sequence models. Key applications include machine translation (text to text), image captioning (pixels to text), and text summarization (long text to short text).
WHEN NOT TO USE IT: It's not used in models that lack a distinct encoder-decoder structure, such as decoder-only language models like GPT. These models generate text based only on the preceding text they've already generated, using self-attention to understand context within that single sequence.
ONE CANONICAL EXAMPLE: In translating "The quick brown fox" to German, when the decoder is generating "schnelle", its query will find high attention scores with the key for "quick". This tells the model to focus on the information from "quick" to produce "schnelle", even though the words are in different positions in their respective sentences.
Read the original → arxiv.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.