Describe Transformer architecture and why self-attention beats recurrence

WHAT THIS TESTS: The interviewer wants to know if you understand why the Transformer replaced recurrence for sequence modeling. Specifically they are checking whether you can articulate the trade-off between sequential computation and parallelization, explain how self-attention changes the path length between any two positions in a sequence, and connect these mechanics to training speed and model quality. They also want to see if you know the high-level building blocks beyond just attention.

A GOOD ANSWER COVERS: First, the encoder-decoder stack structure with N identical layers, each containing multi-head self-attention and a position-wise feed-forward network, plus residual connections and layer normalization. Second, positional encodings added to the input embeddings because the model otherwise has no notion of token order. Third, the role of self-attention as a mechanism that computes query-key-value interactions so every position can attend to every other position in a single operation, creating direct paths between distant tokens. Fourth, the contrast with RNNs and LSTMs which must unroll sequentially in O(n) steps, creating long-range paths of length O(n) that dilute gradients and prevent parallelization across the time dimension. Fifth, the practical advantage that Transformers are more parallelizable and train significantly faster, as shown by the original paper achieving 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French after only 3.5 days on eight GPUs.

COMMON WRONG ANSWERS: A red flag is describing self-attention only as a weighted average of values without explaining why that matters for path length or parallel training. Another mistake is claiming that Transformers have lower memory usage than RNNs; in fact the full attention matrix is quadratic in sequence length. Some candidates also forget positional encodings entirely, implying the model is permutation invariant by design. Confusing self-attention with the older encoder-decoder attention from seq2seq models is also a common error.

LIKELY FOLLOW-UPS: The interviewer may ask how the quadratic complexity of attention limits sequence length and what sparse or linear attention variants do about it. They might probe the difference between self-attention, cross-attention, and masked self-attention in the decoder. Another common follow-up is why layer normalization and residual connections are essential for training deep Transformer stacks, or how positional encodings have evolved into learned positional embeddings and rotary embeddings.

ONE CONCRETE EXAMPLE: If you are translating a long sentence from English to German, an LSTM must process the source tokens one by one and the hidden state at the final token has traveled through dozens of nonlinear transformations, making it hard to retain the subject-verb agreement from the start of the sentence. In a Transformer, the encoder's self-attention layer directly connects the first token to the last token in one matrix multiplication, so the model can preserve that agreement while still training all positions in parallel across the batch.