Cross-Attention: How Models Fuse Text and Images

Why it exists

Models needed a way to combine information from fundamentally different sources, like text and images. Self-attention could only find relationships within a single source, like one block of text. Cross-attention was created to bridge this gap, allowing one modality to "query" another and fuse their representations.

The mental model

Think of cross-attention as a targeted lookup between two different books. You have a concept from Book A (the query, e.g., a text token for "cat") and you need to find the most relevant illustration in Book B (the keys/values, e.g., a library of image patches). Cross-attention finds the best matching patch and blends its information back into your process. It's a mechanism for targeted information retrieval across different contexts.

How it works

Cross-attention uses the same mathematical formula as self-attention: Attention(Q, K, V) = softmax(Q K_transpose / sqrt(d)) V. The crucial difference is the source of the inputs. In self-attention, Q, K, and V are all derived from the same input sequence. In cross-attention, the query matrix Q is derived from one sequence (e.g., text embeddings), while the key (K) and value (V) matrices are derived from a second, different sequence (e.g., image patch embeddings). This decoupling allows the first sequence to selectively integrate information from the second.

When to use it

Use cross-attention when you need to fuse or condition one data stream with another. It is essential for multi-modal tasks. Prime examples include text-to-image generation, where text prompts guide the creation of image pixels; visual question answering, where a question about an image must attend to relevant parts of that image; and video processing, where audio might be aligned with specific visual frames.

When not to use it

Don't use cross-attention when your goal is to understand relationships within a single, homogeneous sequence of data. For tasks like summarizing a block of text or understanding the grammatical structure of a sentence, standard self-attention is the appropriate and more direct tool. Cross-attention introduces complexity that is unnecessary if you aren't working with multiple distinct data sources.

One canonical example

In a text-to-image model like Stable Diffusion, a text prompt is first converted into token embeddings. These text embeddings act as the queries (Q). The model also has a noisy, evolving image representation, which is broken into patches and converted into key (K) and value (V) embeddings. The cross-attention mechanism calculates how much each text token "attends" to each image patch. This allows the model to ensure that the part of the image corresponding to the word "blue" is actually colored blue, and the part corresponding to "cat" looks like a cat.