How Transformers encode token position

WHAT THIS TESTS Whether you understand the permutation invariance of attention and the menu of positional encoding strategies plus their tradeoffs.

A GOOD ANSWER COVERS Self-attention computes weighted sums over all tokens with no notion of order; permuting the inputs permutes the outputs identically, so the model would see a sentence as a bag of tokens. To fix this, position information is injected. The original Transformer adds fixed sinusoidal positional encodings, vectors built from sines and cosines at varying frequencies, to the token embeddings; these need no training and extrapolate somewhat to unseen lengths. An alternative is learned absolute position embeddings, a trainable vector per position, simple but capped at the trained maximum length. Modern models often use relative position schemes, which encode the distance between tokens rather than absolute index, and rotary position embeddings (RoPE), which rotate the Query and Key vectors by position-dependent angles so that dot products naturally reflect relative offset. Relative and rotary methods tend to generalize better to longer contexts.

COMMON WRONG ANSWERS The Transformer processes tokens in order like an RNN. Position is irrelevant because of attention. Confusing positional encoding with token embedding. Thinking sinusoidal encodings are learned, or that learned absolute encodings extrapolate to arbitrary lengths.

LIKELY FOLLOW-UPS Why might sinusoidal generalize to longer sequences than learned absolute? What advantage does RoPE give for long context? How are encodings combined with embeddings, added or concatenated? What breaks when you exceed the trained position range?

ONE CONCRETE EXAMPLE For dog bites man versus man bites dog, the token embeddings are identical sets, so attention alone cannot tell them apart. Adding distinct positional encodings to each slot makes dog at position one differ from dog at position three, letting the model learn that subject and object roles depend on order. With RoPE, the Query-Key dot product for adjacent versus distant tokens differs by construction, encoding relative distance directly.