How Transformers encode token position
Why and how positional information is injected.
Attention is permutation-invariant, so positional encodings (sinusoidal, learned, or rotary) are added or applied.
What's really being asked
Whether you understand the permutation invariance of attention and the menu of positional encoding strategies plus their tradeoffs.
The full answer
Self-attention computes weighted sums over all tokens with no notion of order; permuting the inputs permutes the outputs identically, so the model would see a sentence as a bag of tokens. To fix this, position information is injected. The original Transformer adds fixed sinusoidal positional encodings, vectors built from sines and cosines at varying frequencies, to the token embeddings; these need no training and extrapolate somewhat to unseen lengths. An alternative is learned absolute position embeddings, a trainable vector per position, simple but capped at the trained maximum length. Modern models often use relative position schemes, which encode the distance between tokens rather than absolute index, and rotary position embeddings (RoPE), which rotate the Query and Key vectors by position-dependent angles so that dot products naturally reflect relative offset. Relative and rotary methods tend to generalize better to longer contexts.
The mistakes people make
The Transformer processes tokens in order like an RNN. Position is irrelevant because of attention. Confusing positional encoding with token embedding. Thinking sinusoidal encodings are learned, or that learned absolute encodings extrapolate to arbitrary lengths.
What usually comes next
Why might sinusoidal generalize to longer sequences than learned absolute? What advantage does RoPE give for long context? How are encodings combined with embeddings, added or concatenated? What breaks when you exceed the trained position range?
A concrete example
For dog bites man versus man bites dog, the token embeddings are identical sets, so attention alone cannot tell them apart. Adding distinct positional encodings to each slot makes dog at position one differ from dog at position three, letting the model learn that subject and object roles depend on order. With RoPE, the Query-Key dot product for adjacent versus distant tokens differs by construction, encoding relative distance directly.
Interview question
Why must Transformers add positional encodings, whereas RNNs do not need them?
- a.RNNs are also permutation-invariant but ignore the problem
- b.Attention already encodes absolute position in its weights
- c.Positional encodings replace the need for token embeddings
- d.Self-attention is permutation-invariant, so without added position signals it cannot distinguish token orderCorrect
Why? this is the answer
Attention treats inputs as an unordered set, so order must be injected explicitly. RNNs process tokens sequentially, giving them order implicitly; attention has no such built-in ordering.
Just read this? Test yourself on what you have been reading.
- #llm
- #transformer
- #positional-encoding
- #rope
- #architecture
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles