Encoder-Only vs. Decoder-Only vs. Encoder-Decoder Transformers?
This tests your ability to connect transformer architecture to specific NLP tasks. A great answer explains how each model's attention mechanism dictates its use: encoder-only (bidirectional attention) for understanding content, decoder-only (causal attention) for text generation, and encoder-decoder for sequence-to-sequence tasks like translation. The key red flag is failing to explain the *why* behind the task suitability—the attention mechanism.
What's really being asked
This tests your understanding of how a transformer's internal architecture, specifically its attention mechanism, determines its suitability for different NLP tasks. The interviewer wants to see if you can connect the 'how' (bidirectional vs. causal attention) to the 'what' (classification, generation, translation).
The full answer
• Encoder-Only (e.g., BERT, RoBERTa): Uses bidirectional (unmasked) self-attention. Each token can attend to all other tokens in the sequence. This provides deep contextual understanding of the entire input. Best for Natural Language Understanding (NLU) tasks like sentiment analysis, named entity recognition, and sentence classification. • Decoder-Only (e.g., GPT series, Llama): Uses causal (or auto-regressive) masked self-attention. Each token can only attend to preceding tokens and itself. This structure is inherently designed for next-token prediction, making it perfect for free-form text generation. • Encoder-Decoder (e.g., T5, BART, original "Attention Is All You Need" paper): A two-part model. The encoder uses bidirectional attention on a source sequence (e.g., an English sentence). The decoder uses causal attention on the target sequence (e.g., a French sentence) but also performs cross-attention to the encoder's output. This makes it ideal for sequence-to-sequence tasks like machine translation, summarization, and question answering.
The mistakes people make
• Simply listing examples ("BERT is an encoder, GPT is a decoder") without explaining the architectural difference in their attention mechanisms. • Incorrectly describing the attention flow, for example, saying encoders use causal attention or that decoders can see the full future sequence. • Confusing the model families, e.g., claiming GPT is an encoder-decoder model.
Follow-up the interviewer might ask
• "Why would you choose a model like BART (an encoder-decoder) over GPT (a decoder-only) for summarization?" • "Could you use a decoder-only model for classification? How would you set that up?"
A concrete example
"For a sentiment analysis task on 10,000 movie reviews, I'd choose a fine-tuned BERT (encoder-only) model. Its bidirectional attention allows it to understand context from the entire review, like how a 'not' at the beginning negates a 'good' at the end. Using a GPT model (decoder-only) would be inefficient and less effective, as its causal attention is designed for generating text, not classifying a fixed input."
Interview question
Which transformer architecture is best suited for machine translation from English to French, and why?
- a.A. Encoder-Only: Because its bidirectional attention allows for a comprehensive understanding of the entire English input sentence.
- b.B. Decoder-Only: Because its causal attention mechanism is optimized for generating the French output sequence token by token.
- c.C. Encoder-Decoder: Because its encoder processes the English input bidirectionally, and its decoder generates the French output causally while leveraging cross-attention to the encoder's understanding.Correct
- d.D. Encoder-Decoder: Because it uses bidirectional attention for both the English input and the French output, ensuring full context for both languages.
Why? this is the answer
The Encoder-Decoder architecture is ideal for sequence-to-sequence tasks like machine translation. Its encoder uses bidirectional attention to fully understand the source language, and its decoder generates the target language causally, critically using cross-attention to integrate the encoder's contextual understanding. Option B is a tempting distractor because decoder-only models generate text, but they lack the dedicated bidirectional source understanding and cross-attention mechanism crucial for effective translation.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles