How does positional encoding work in transformers?
This tests your understanding of why Transformers need explicit position data. A great answer explains that self-attention is permutation-invariant, meaning it sees inputs as an unordered set. Positional encodings—vectors derived from sine and cosine functions—are then added to the input embeddings to inject sequence order. A red flag is simply saying 'it adds position' without explaining why this is necessary or how it's done.
What's really being asked
This tests your grasp of a fundamental limitation of the self-attention mechanism. Because attention processes all tokens in parallel, it is 'permutation-invariant'—it has no built-in sense of sequence order. An interviewer wants to see that you understand this core problem and can articulate the specific mechanism used to solve it.
The full answer
• The Problem: Start by stating that the self-attention mechanism is permutation-invariant. Without positional information, the model would treat "man bites dog" and "dog bites man" as identical sets of words.
• The High-Level Solution: To solve this, a 'positional encoding' vector is created for each position in the input sequence (e.g., position 0, 1, 2, ... N).
• The Mechanism: This positional encoding vector has the same dimension as the token embeddings (e.g., 512). It is added to the corresponding token's embedding vector. The resulting vector now contains information about both the token's semantic meaning and its absolute position in the sequence.
• The Original Implementation: The "Attention Is All You Need" paper used a fixed (not learned) method with sine and cosine functions of varying frequencies. This choice has the useful property that the encoding for position pos+k can be represented as a linear function of the encoding for pos, which helps the model learn relative positions.
The mistakes people make
Vague Hand-waving: "It just adds a number to tell the model the position." This misses the why (permutation invariance) and the how* (vector addition, sine/cosine functions). Confusing it with Token Embeddings: Stating that the positional encoding is* the embedding. They are two separate vectors that are combined. Incorrect Combination: Saying the positional encoding is concatenated with the token embedding. While a possible design, the canonical Transformer architecture adds* them.
Follow-up the interviewer might ask
• "Why use sine and cosine? Why not just use integers 0, 1, 2...?" • "Are there alternatives to this fixed encoding scheme?"
A concrete example
For a sentence "Hello world" with a model dimension of 4:
1. Token Embeddings:
* Hello -> E_hello = [0.1, 0.8, 0.2, 0.9]
* world -> E_world = [0.5, 0.2, 0.6, 0.3]
2. Positional Encodings (calculated via sin/cos):
* Position 0 -> P_0 = [0.0, 1.0, 0.0, 1.0]
* Position 1 -> P_1 = [0.84, 0.54, 0.01, 0.99]
3. Input to Transformer:
* Input 0 = E_hello + P_0 = [0.1, 1.8, 0.2, 1.9]
* Input 1 = E_world + P_1 = [1.34, 0.74, 0.61, 1.29]
Interview question
Why are positional encodings essential for the Transformer's self-attention mechanism?
- a.They inject sequence order information, which self-attention inherently lacks due to its permutation-invariant nature.Correct
- b.They replace the original token embeddings with position-aware representations, enhancing semantic meaning.
- c.They increase the overall dimensionality of the input vectors, allowing for more complex pattern recognition.
- d.They concatenate position-specific features to the token embeddings, providing additional context for each word.
Why? this is the answer
The self-attention mechanism is permutation-invariant, meaning it processes tokens as an unordered set. Positional encodings are added to token embeddings specifically to inject sequence order, allowing the model to understand the relative positions of words. Option D is incorrect because positional encodings are added, not concatenated, and their purpose is specifically to convey order, not just 'additional context'.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #llm
- #transformer
- #architecture
- #attention
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles