Transformer Encoder-Decoder Architecture

WHY IT EXISTS Many tasks transform one variable-length sequence into another: translating English to French, summarizing an article, transcribing speech. Recurrent encoder-decoders squeezed the entire source into a single fixed vector, which lost detail for long inputs and could not be parallelized. The Transformer encoder-decoder was designed to read the whole source in parallel and let the generator look back at any part of it.

THE MENTAL MODEL Think of two cooperating stacks. The encoder is a reader that builds a complete, context-rich understanding of the input. The decoder is a writer that produces the output token by token, and at every step it can consult both what it has written so far and the encoder's full understanding of the source.

HOW IT WORKS The encoder applies bidirectional self-attention so each input token attends to all others, then feed-forward layers, repeated across stacked blocks, yielding one representation per input token. The decoder uses three sublayers per block: masked self-attention that prevents a position from seeing future tokens during training, cross-attention where the decoder's queries attend to the encoder's keys and values, and a feed-forward layer. Generation is autoregressive: each predicted token is fed back in to predict the next.

WHEN IT MATTERS This architecture shines for sequence-to-sequence problems where input and output are distinct sequences. It powers translation models and is the basis of T5 and BART. Encoder-only models like BERT suit understanding tasks, and decoder-only models like GPT suit open-ended generation, but the encoder-decoder is the natural fit when a full source conditions the output.

ONE CONCRETE EXAMPLE Translating the English sentence into French, the encoder processes all English tokens simultaneously and outputs a vector per word capturing meaning in context. The decoder then emits French tokens one by one. When generating each French word, its cross-attention weights highlight the relevant English words, so producing the French verb draws on the corresponding English verb, while masked self-attention keeps it from peeking at French words it has not yet generated.