Why ViTs need positional embeddings

WHAT THIS TESTS Whether you understand that self-attention is permutation invariant while convolution is spatially grounded, and can name a concrete positional-embedding scheme.

A GOOD ANSWER COVERS Self-attention computes each output as a weighted combination of all tokens based purely on their content-derived queries and keys, with no notion of order. If you permute the input patch tokens, the set of outputs is permuted identically; the operation treats its input as an unordered set. So a plain ViT cannot distinguish a patch in the top-left from one in the bottom-right, yet spatial arrangement is essential for images. Positional embeddings fix this by adding a unique, position-dependent vector to each patch embedding before the transformer, so each token carries where it came from in addition to what it contains. A CNN needs no such mechanism because position is implicit in its architecture: filters slide over a fixed spatial grid and feature maps preserve the layout, so a feature's location in the map directly encodes its location in the image.

ONE IMPLEMENTATION The original ViT uses learnable absolute positional embeddings: a separate trainable vector per patch position, added to the corresponding patch token and learned end to end. An alternative is fixed sinusoidal encodings, where each position gets a deterministic vector built from sines and cosines of varying frequencies, requiring no parameters and extrapolating somewhat to unseen lengths. Relative position embeddings, used in Swin, encode the offset between patch pairs inside the attention computation.

COMMON WRONG ANSWERS Saying attention inherently knows positions. Claiming CNNs also need explicit positional embeddings. Confusing positional embeddings with the class token.

LIKELY FOLLOW-UPS Learnable versus sinusoidal trade-offs? How do you handle a different image resolution at test time, interpolating the embeddings? What are relative and rotary positional embeddings? Does adding versus concatenating position matter?

ONE CONCRETE EXAMPLE Shuffle the 196 patches of an image fed to a ViT with no positional embeddings and the prediction is unchanged, proving position blindness; add learnable positional embeddings and the same shuffle now changes the output, because each token knows its grid location.