Self-Attention versus Recurrent Architectures
understanding self-attention and its edge over RNNs.
each token attends to all others via query-key-value, enabling parallelism and direct long-range links.
WHAT THIS TESTS This evaluates whether you can explain self-attention intuitively and articulate the concrete advantages that let Transformers replace recurrent models.
A GOOD ANSWER COVERS In self-attention every token in a sequence builds three vectors from learned projections: a query, a key, and a value. To compute a token's new representation, its query is compared against the keys of all tokens, including itself, producing similarity scores. These scores are scaled and passed through a softmax to become weights, and the output is the weighted sum of all tokens' value vectors. The result is that each token's representation is a blend of the whole sequence, weighted by relevance, so any token can directly attend to any other in a single step.
WHY IT BEAT RECURRENCE LSTMs process tokens sequentially, so training cannot parallelize across the time dimension, and information from distant tokens must survive many sequential steps, which weakens long-range dependencies. Self-attention computes all pairwise interactions at once, enabling massive parallelism on GPUs and a constant path length between any two positions, which makes learning long-range relationships far easier and training much faster.
COMMON WRONG ANSWERS Confusing self-attention with the decoder-to-encoder cross-attention of older seq2seq models. Forgetting positional encodings, since attention alone is order-agnostic. Overstating that attention is always cheaper; it is quadratic in sequence length, a real cost for long inputs.
LIKELY FOLLOW-UPS Why divide the scores by the square root of the key dimension? Why are positional encodings needed? What is the time and memory complexity in sequence length? How does multi-head attention extend the single-head idea?
ONE CONCRETE EXAMPLE In the sentence about an animal that did not cross the street because it was too tired, resolving what it refers to requires linking a pronoun to a noun several words earlier. An LSTM must carry that information forward step by step and may lose it. Self-attention lets the pronoun's query directly score highly against the relevant noun's key in one operation, so the model resolves the reference immediately regardless of the gap.
Interview question
Beyond modeling long-range dependencies, what key training advantage does self-attention have over an LSTM?
- a.It requires no positional information because order is built in
- b.It eliminates the need for any learned weight matrices
- c.It can process all sequence positions in parallel rather than step by stepCorrect
- d.It has lower computational complexity in sequence length than recurrence
Why? this is the answer
Self-attention computes all token interactions simultaneously, enabling parallel training that sequential LSTMs cannot match. It actually needs added positional encodings and is quadratic, not cheaper, in sequence length.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #self-attention
- #transformers
- #lstm
- #parallelism
- #nlp
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles