Cross-attention for visual question answering

WHAT THIS TESTS Whether you understand that cross-attention lets one modality query another, and can wire it into a concrete VQA pipeline.

A GOOD ANSWER COVERS Self-attention computes queries, keys, and values all from the same sequence. Cross-attention breaks that symmetry: the queries come from one modality and the keys and values come from another, so one stream attends to and pulls information from the other. For visual question answering, first encode each modality separately. The image goes through a vision backbone, a CNN or ViT, producing a set of region or patch feature vectors. The question goes through a text encoder producing token embeddings. Then apply cross-attention so that the text tokens act as queries against the image features as keys and values. Each word's query scores against every image region; the softmax weights let that word gather the visual regions most relevant to it, for example the word color attending to the object being asked about. The attended visual context is fused with the language representation, optionally with additional layers attending the other direction too, and a classification head over a fixed answer vocabulary, or a decoder, produces the answer. Models like LXMERT, ViLBERT, and BLIP use exactly this co-attention pattern.

COMMON WRONG ANSWERS Calling it self-attention where everything comes from one sequence. Fusing by simple concatenation with no attention, which cannot align words to regions. Letting image attend to itself only, missing the cross-modal link.

LIKELY FOLLOW-UPS Why is cross-attention better than late fusion by concatenation? How does bidirectional co-attention help? How are answers represented, classification versus generation? How do you handle alignment when region and token counts differ?

ONE CONCRETE EXAMPLE For the question what color is the bus, the bus token's query attends across image regions, softmax weights concentrate on the bus region's value vectors, and the fused feature, dominated by that region, lets the classifier output red rather than attending to irrelevant sky regions.