How does positional encoding work in transformers?
This tests your understanding of why Transformers need explicit position data. A great answer explains that self-attention is permutation-invariant, meaning it sees inputs as an unordered set. Positional encodings—vectors derived from sine and cosine functions—are then added to the input embeddings to inject sequence order. A red flag is simply saying 'it adds position' without explaining why this is necessary or how it's done.
### What this tests This tests your grasp of a fundamental limitation of the self-attention mechanism. Because attention processes all tokens in parallel, it is 'permutation-invariant'—it has no built-in sense of sequence order. An interviewer wants to see that you understand this core problem and can articulate the specific mechanism used to solve it.
### A good answer covers * **The Problem:** Start by stating that the self-attention mechanism is permutation-invariant. Without positional information, the model would treat "man bites dog" and "dog bites man" as identical sets of words. * **The High-Level Solution:** To solve this, a 'positional encoding' vector is created for each position in the input sequence (e.g., position 0, 1, 2, ... N). * **The Mechanism:** This positional encoding vector has the same dimension as the token embeddings (e.g., 512). It is **added** to the corresponding token's embedding vector. The resulting vector now contains information about both the token's semantic meaning and its absolute position in the sequence. * **The Original Implementation:** The "Attention Is All You Need" paper used a fixed (not learned) method with sine and cosine functions of varying frequencies. This choice has the useful property that the encoding for position `pos+k` can be represented as a linear function of the encoding for `pos`, which helps the model learn relative positions.
### Common wrong answers * **Vague Hand-waving:** "It just adds a number to tell the model the position." This misses the *why* (permutation invariance) and the *how* (vector addition, sine/cosine functions). * **Confusing it with Token Embeddings:** Stating that the positional encoding *is* the embedding. They are two separate vectors that are combined. * **Incorrect Combination:** Saying the positional encoding is concatenated with the token embedding. While a possible design, the canonical Transformer architecture *adds* them.
### Follow-up the interviewer might ask * "Why use sine and cosine? Why not just use integers 0, 1, 2...?" * "Are there alternatives to this fixed encoding scheme?"
### One concrete example For a sentence "`Hello world`" with a model dimension of 4: 1. **Token Embeddings**: * `Hello` -> `E_hello` = `[0.1, 0.8, 0.2, 0.9]` * `world` -> `E_world` = `[0.5, 0.2, 0.6, 0.3]` 2. **Positional Encodings** (calculated via sin/cos): * `Position 0` -> `P_0` = `[0.0, 1.0, 0.0, 1.0]` * `Position 1` -> `P_1` = `[0.84, 0.54, 0.01, 0.99]` 3. **Input to Transformer**: * `Input 0` = `E_hello + P_0` = `[0.1, 1.8, 0.2, 1.9]` * `Input 1` = `E_world + P_1` = `[1.34, 0.74, 0.61, 1.29]`
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.