Cross-attention in transformer VQA models

WHAT THIS TESTS The question checks that you understand multimodal fusion as a learned, query-driven interaction rather than simple feature stacking. VQA needs the model to connect words to image regions.

A GOOD ANSWER COVERS Mechanics: attention computes a weighted sum of values using query-key similarity. In cross-attention the queries are derived from one modality, say text token embeddings, while the keys and values come from the other, the image region or patch features. Each text token thus attends over all visual features, producing a context vector that pulls in the most relevant visual evidence. Stacked layers refine this. What it learns: a soft alignment or grounding between language and vision, so the word referring to an object aligns with the image patches containing it, letting the model reason about attributes, relations, and counts to produce an answer.

COMMON WRONG ANSWERS Describing self-attention, where queries, keys, and values all come from the same modality, and calling it cross-attention. Or saying fusion is just concatenating image and text vectors, which lacks the dynamic, content-dependent weighting. Or claiming attention is hard-coded rather than learned end to end.

LIKELY FOLLOW-UPS What is the difference between self-attention and cross-attention. How are visual features tokenized, as region proposals or patches. How does multi-head attention help capture different relations. How would you visualize the learned grounding.

ONE CONCRETE EXAMPLE For the question what color is the umbrella, the token for umbrella forms a query that attends strongly to the image patches containing the umbrella while down-weighting the background and other objects. The resulting context vector encodes that region's appearance, and the answer head reads off the color, demonstrating learned word-to-region grounding. Multiple attention heads let the model attend to different aspects at once, for instance one head locating the umbrella and another relating it to the person holding it, which supports reasoning about relations and counts rather than just isolated objects.