Cross-attention versus self-attention in encoder-decoder Transformers
Information flow in encoder-decoder models.
Cross-attention draws Queries from the decoder and Keys/Values from the encoder, letting the decoder condition on the source.
What's really being asked
Whether you can pinpoint the source of Q, K, and V in each attention block and explain why cross-attention is the bridge between encoder and decoder.
The full answer
There are three attention blocks in a classic encoder-decoder. Encoder self-attention lets each source token attend to all other source tokens; Q, K, and V all come from the source, with no masking, so it is bidirectional. Decoder self-attention lets each target token attend to previously generated target tokens; Q, K, and V come from the target sequence, but a causal mask blocks attending to future positions so generation stays autoregressive. Cross-attention is the distinguishing piece: it lives in each decoder layer between the decoder self-attention and the feed-forward sublayer. Its Queries come from the decoder's current hidden states, while its Keys and Values come from the encoder's final output. This lets every decoder position look back over the entire encoded source, which is how a translation decoder aligns each output word with relevant source words. It is not causally masked over the source, since the whole source is available.
The mistakes people make
Saying cross-attention is the same as self-attention. Claiming all of Q, K, V come from the decoder. Applying a causal mask to the encoder Keys. Forgetting that decoder self-attention is masked while cross-attention is not. Putting cross-attention in the encoder.
What usually comes next
Why is decoder self-attention masked but cross-attention is not? Where does cross-attention sit within a decoder layer? Why do decoder-only models lack cross-attention? How does cross-attention enable alignment in translation?
A concrete example
Translating the house is red to French, the encoder produces hidden states for the four English tokens. As the decoder generates rouge, its cross-attention Query attends over the encoder Keys and Values and places high weight on the encoder state for red, pulling that meaning into the output, while decoder self-attention separately tracks the French words generated so far.
Interview question
In cross-attention within an encoder-decoder Transformer, where do the Query, Key, and Value vectors originate?
- a.Queries from the encoder; Keys and Values from the decoder
- b.Queries, Keys, and Values all from the encoder output
- c.Queries from the decoder; Keys and Values from the encoder outputCorrect
- d.Queries, Keys, and Values all from the decoder
Why? this is the answer
Cross-attention uses decoder hidden states as Queries and encoder outputs as Keys and Values, letting the decoder condition on the source. If all came from one stack it would just be self-attention.
Just read this? Test yourself on what you have been reading.
- #llm
- #transformer
- #cross-attention
- #encoder-decoder
- #architecture
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles