Encoder, decoder, and encoder-decoder Transformers
mapping architecture to objective and use case.
encoder-only uses bidirectional masked-token pretraining for understanding tasks, decoder-only uses causal next-token prediction for generation, encoder-decoder uses span corruption for…
What's really being asked
This evaluates whether you understand that attention masking and pretraining objective determine what each Transformer family is good at, and can match each to its natural tasks.
The full answer
Encoder-only models like BERT use bidirectional self-attention, so every token attends to the full sequence on both sides. They are pretrained with masked language modeling, predicting randomly hidden tokens from surrounding context, which forces deep bidirectional understanding. This makes them ideal for tasks that consume text and output labels or spans: classification, sentiment, named-entity recognition, and producing embeddings for retrieval. They are not built to generate fluent text. Decoder-only models like GPT use causal, left-to-right masking so a token sees only earlier tokens, and are pretrained on next-token prediction. This autoregressive objective makes them natural generators for chat, completion, and few-shot prompting. Encoder-decoder models like T5 combine both: a bidirectional encoder builds a representation of the input, and an autoregressive decoder with cross-attention generates the output. T5 is pretrained with span corruption, masking contiguous spans and reconstructing them, and frames every problem as text-to-text, making it strong for translation, summarization, and question answering where input and output are both sequences.
The mistakes people make
Saying BERT generates text autoregressively; calling GPT bidirectional; confusing span corruption with simple masked LM; ignoring cross-attention in encoder-decoder; assuming one architecture is best for everything.
What usually comes next
Why does causal masking enable generation? How does cross-attention work in T5? Why is BERT poor at generation? Could a decoder-only model do classification?
A concrete example
For extracting entities from contracts you fine-tune BERT, whose bidirectional context pinpoints span boundaries. For a customer-facing chatbot you use a GPT-style decoder that continues text token by token. For translating documents you use T5, whose encoder reads the source bidirectionally and decoder emits the target sequence.
Interview question
Which pretraining objective and masking scheme makes decoder-only models like GPT naturally suited to text generation?
- a.Bidirectional attention with masked language modeling
- b.Span corruption with cross-attention to an encoder
- c.Contrastive learning over sentence pairs
- d.Causal left-to-right masking with next-token predictionCorrect
Why? this is the answer
Decoder-only models mask future tokens and learn next-token prediction, exactly the autoregressive setup generation needs. Bidirectional masked LM describes BERT, and span corruption with cross-attention describes T5.
Just read this? Test yourself on what you have been reading.
Read the original → huggingface.co
- #transformers
- #bert
- #gpt
- #t5
- #architecture
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on transformers — each one lists the topics its interview covers.
See open roles