Self-Attention versus Recurrent Architectures

WHAT THIS TESTS This evaluates whether you can explain self-attention intuitively and articulate the concrete advantages that let Transformers replace recurrent models.

A GOOD ANSWER COVERS In self-attention every token in a sequence builds three vectors from learned projections: a query, a key, and a value. To compute a token's new representation, its query is compared against the keys of all tokens, including itself, producing similarity scores. These scores are scaled and passed through a softmax to become weights, and the output is the weighted sum of all tokens' value vectors. The result is that each token's representation is a blend of the whole sequence, weighted by relevance, so any token can directly attend to any other in a single step.

WHY IT BEAT RECURRENCE LSTMs process tokens sequentially, so training cannot parallelize across the time dimension, and information from distant tokens must survive many sequential steps, which weakens long-range dependencies. Self-attention computes all pairwise interactions at once, enabling massive parallelism on GPUs and a constant path length between any two positions, which makes learning long-range relationships far easier and training much faster.

COMMON WRONG ANSWERS Confusing self-attention with the decoder-to-encoder cross-attention of older seq2seq models. Forgetting positional encodings, since attention alone is order-agnostic. Overstating that attention is always cheaper; it is quadratic in sequence length, a real cost for long inputs.

LIKELY FOLLOW-UPS Why divide the scores by the square root of the key dimension? Why are positional encodings needed? What is the time and memory complexity in sequence length? How does multi-head attention extend the single-head idea?

ONE CONCRETE EXAMPLE In the sentence about an animal that did not cross the street because it was too tired, resolving what it refers to requires linking a pronoun to a noun several words earlier. An LSTM must carry that information forward step by step and may lose it. Self-attention lets the pronoun's query directly score highly against the relevant noun's key in one operation, so the model resolves the reference immediately regardless of the gap.