Transformer Encoder-Decoder Architecture
The encoder-decoder Transformer maps an input sequence into rich contextual representations with an encoder, then a decoder generates output tokens autoregressively while attending to those representations via cross-attention, making it ideal for…
WHY IT EXISTS Many tasks transform one variable-length sequence into another: translating English to French, summarizing an article, transcribing speech. Recurrent encoder-decoders squeezed the entire source into a single fixed vector, which lost detail for long inputs and could not be parallelized. The Transformer encoder-decoder was designed to read the whole source in parallel and let the generator look back at any part of it.
THE MENTAL MODEL Think of two cooperating stacks. The encoder is a reader that builds a complete, context-rich understanding of the input. The decoder is a writer that produces the output token by token, and at every step it can consult both what it has written so far and the encoder's full understanding of the source.
HOW IT WORKS The encoder applies bidirectional self-attention so each input token attends to all others, then feed-forward layers, repeated across stacked blocks, yielding one representation per input token. The decoder uses three sublayers per block: masked self-attention that prevents a position from seeing future tokens during training, cross-attention where the decoder's queries attend to the encoder's keys and values, and a feed-forward layer. Generation is autoregressive: each predicted token is fed back in to predict the next.
WHEN IT MATTERS This architecture shines for sequence-to-sequence problems where input and output are distinct sequences. It powers translation models and is the basis of T5 and BART. Encoder-only models like BERT suit understanding tasks, and decoder-only models like GPT suit open-ended generation, but the encoder-decoder is the natural fit when a full source conditions the output.
ONE CONCRETE EXAMPLE Translating the English sentence into French, the encoder processes all English tokens simultaneously and outputs a vector per word capturing meaning in context. The decoder then emits French tokens one by one. When generating each French word, its cross-attention weights highlight the relevant English words, so producing the French verb draws on the corresponding English verb, while masked self-attention keeps it from peeking at French words it has not yet generated.
Interview question
In a Transformer encoder-decoder, what is the specific role of the decoder's cross-attention sublayer?
- a.It lets each decoder position attend to the encoder's output representations of the inputCorrect
- b.It prevents the decoder from attending to future output tokens during training
- c.It normalizes the decoder activations between feed-forward layers
- d.It applies bidirectional attention across all input tokens in the encoder
Why? this is the answer
Cross-attention is where decoder queries attend to the encoder's keys and values, conditioning generation on the source. Blocking future tokens is masked self-attention's job, and bidirectional input attention happens in the encoder.
Just read this? Test yourself on what you have been reading.
- #transformers
- #encoder-decoder
- #attention
- #seq2seq
- #nlp
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on transformers — each one lists the topics its interview covers.
See open roles