Attention in Sequence-to-Sequence Models

WHAT THIS TESTS This checks whether you understand the motivation for attention, not just its formula, by tying it to the concrete weakness of fixed-vector encoder-decoders.

A GOOD ANSWER COVERS A classic RNN encoder-decoder reads the input and compresses everything into a single fixed-size context vector, the final hidden state, which the decoder then unrolls. This is an information bottleneck: one vector cannot faithfully hold all the detail of a long sentence, so quality degrades as inputs grow, and early input information is forgotten. Attention removes the bottleneck. At each decoding step the decoder compares its current state to every encoder hidden state to produce alignment scores, passes them through a softmax to get weights, and forms a context vector as the weighted sum of encoder states. So the decoder dynamically attends to the most relevant input positions for the token it is generating right now, and gradients also flow more directly to the relevant encoder states.

COMMON WRONG ANSWERS Describing the weighted-sum mechanics but never naming the fixed-vector bottleneck it solves. Confusing this additive seq2seq attention with Transformer self-attention; here the decoder attends to encoder states, not tokens attending among themselves. Claiming attention removes the need for the encoder.

LIKELY FOLLOW-UPS What is the difference between additive Bahdanau and multiplicative Luong attention? How do alignment weights give interpretability? How does this generalize to the scaled dot-product self-attention of Transformers? What is the computational cost relative to sequence length?

ONE CONCRETE EXAMPLE Translating a long English sentence to French, a vanilla model crams the whole sentence into one vector and tends to mistranslate the end of long inputs. With attention, when the decoder generates the French word for a subject noun appearing late in the English source, the alignment weights peak on that specific English word, letting the model pull its meaning directly regardless of sentence length, which sharply improves long-sentence translation.