Encoder-Only vs. Decoder-Only vs. Encoder-Decoder Transformers?

### What this tests This tests your understanding of how a transformer's internal architecture, specifically its attention mechanism, determines its suitability for different NLP tasks. The interviewer wants to see if you can connect the 'how' (bidirectional vs. causal attention) to the 'what' (classification, generation, translation).

### A good answer covers * **Encoder-Only (e.g., BERT, RoBERTa):** Uses bidirectional (unmasked) self-attention. Each token can attend to all other tokens in the sequence. This provides deep contextual understanding of the entire input. Best for Natural Language Understanding (NLU) tasks like sentiment analysis, named entity recognition, and sentence classification. * **Decoder-Only (e.g., GPT series, Llama):** Uses causal (or auto-regressive) masked self-attention. Each token can only attend to preceding tokens and itself. This structure is inherently designed for next-token prediction, making it perfect for free-form text generation. * **Encoder-Decoder (e.g., T5, BART, original "Attention Is All You Need" paper):** A two-part model. The encoder uses bidirectional attention on a source sequence (e.g., an English sentence). The decoder uses causal attention on the target sequence (e.g., a French sentence) but also performs cross-attention to the encoder's output. This makes it ideal for sequence-to-sequence tasks like machine translation, summarization, and question answering.

### Common wrong answers * Simply listing examples ("BERT is an encoder, GPT is a decoder") without explaining the architectural difference in their attention mechanisms. * Incorrectly describing the attention flow, for example, saying encoders use causal attention or that decoders can see the full future sequence. * Confusing the model families, e.g., claiming GPT is an encoder-decoder model.

### Follow-up the interviewer might ask * "Why would you choose a model like BART (an encoder-decoder) over GPT (a decoder-only) for summarization?" * "Could you use a decoder-only model for classification? How would you set that up?"

### One concrete example "For a sentiment analysis task on 10,000 movie reviews, I'd choose a fine-tuned BERT (encoder-only) model. Its bidirectional attention allows it to understand context from the entire review, like how a 'not' at the beginning negates a 'good' at the end. Using a GPT model (decoder-only) would be inefficient and less effective, as its causal attention is designed for generating text, not classifying a fixed input."