Self-attention over image patches explained

WHAT THIS TESTS Whether you understand self-attention operationally, what query, key, and value mean, and the exact computation that yields a single patch's output.

A GOOD ANSWER COVERS After patch embedding, each patch is a token vector. Self-attention first projects every token through three separate learned weight matrices into a query, a key, and a value. Intuitively the query represents what this patch is looking for, the key represents what each patch offers as a label of its content, and the value is the actual information a patch will contribute if attended to. To compute the output for one patch, take that patch's query and compute its dot product with the key of every patch including itself, producing a raw similarity score per patch, measuring relevance. These scores are divided by the square root of the key dimension to stabilize gradients, then passed through a softmax so they become non-negative weights summing to one. Finally the output is the weighted sum of all patches' value vectors using those weights. Thus each patch's new representation is a content-based mixture of information from across the whole image, giving global receptive field in one layer. Multi-head attention runs several such projections in parallel.

COMMON WRONG ANSWERS Swapping the roles, for example using values to compute similarity. Forgetting the scaling and softmax, which normalize the weights. Thinking attention only looks at neighboring patches, when it is global.

LIKELY FOLLOW-UPS Why scale by the square root of the dimension? What do multiple heads buy you? Why is attention quadratic in the patch count? How does the class token aggregate information?

ONE CONCRETE EXAMPLE For a patch on a dog's ear, its query scores high against keys of other dog patches and low against sky patches; after softmax those dog patches get large weights, so the ear patch's output is a weighted blend dominated by the dog's values, encoding that it belongs to a coherent object.