Explain positional encodings in Transformers and their necessity

WHAT THIS TESTS: This question tests whether you understand the fundamental limitation of self-attention as a set operation and how Transformer architectures compensate for it. Interviewers want to see that you recognize self-attention is permutation-invariant by design, meaning shuffling input tokens does not change the attention weights between them without additional mechanisms. They are also checking if you can distinguish between absolute and relative positional representations and explain why recurrence or convolution are absent in the original Transformer encoder and decoder stacks.

A GOOD ANSWER COVERS: A strong answer hits four things in order. First, it states clearly that self-attention computes pairwise interactions between tokens using dot-product similarity, which is inherently unordered; swapping two input embeddings leaves their attention score unchanged. Second, it explains that positional encodings are added to the input token embeddings before the first attention layer to provide a unique signature for each position in the sequence. Third, it describes a common implementation such as sinusoidal encodings where each dimension uses a sine or cosine function of a different frequency, allowing the model to learn to attend to relative positions through fixed periodic patterns; alternatively, it mentions learned positional embedding vectors as seen in BERT and other variants. Fourth, it notes that in the original Transformer these encodings are added rather than concatenated, preserving dimensionality and allowing the model to jointly attend to semantic content and position.

COMMON WRONG ANSWERS: The biggest red flag is claiming that the feed-forward network or multi-head attention somehow infers sequence order implicitly during training, making positional information optional. Another weak response is confusing positional encodings with token embeddings or suggesting that padding masks provide position information. Some candidates also incorrectly state that sinusoidal encodings are learned rather than fixed, or that concatenation is the standard fusion method in the original architecture.

LIKELY FOLLOW-UPS: An interviewer might push you to compare absolute versus relative positional encodings, such as those used in RoPE or ALiBi. They could ask why sinusoidal functions were chosen over simple integer indices, which tests whether you understand the need for bounded values and smooth interpolation to unseen sequence lengths. Another common follow-up is how positional information flows through residual connections and layer normalization across depth.

ONE CONCRETE EXAMPLE: In the original Transformer base model with dimension 512, the positional encoding for position pos and dimension i uses sine when i is even and cosine when i is odd, with wavelength increasing across dimensions. This means position 7 receives a unique 512-dimensional vector that is added to the word embedding for the token at that position before entering the first multi-head attention block. Because the pattern is periodic and deterministic, the model can generalize to sequence lengths slightly beyond those seen during training, unlike learned embeddings which are capped at the maximum trained length.