Encoder, decoder, and encoder-decoder Transformers

WHAT THIS TESTS This evaluates whether you understand that attention masking and pretraining objective determine what each Transformer family is good at, and can match each to its natural tasks.

A GOOD ANSWER COVERS Encoder-only models like BERT use bidirectional self-attention, so every token attends to the full sequence on both sides. They are pretrained with masked language modeling, predicting randomly hidden tokens from surrounding context, which forces deep bidirectional understanding. This makes them ideal for tasks that consume text and output labels or spans: classification, sentiment, named-entity recognition, and producing embeddings for retrieval. They are not built to generate fluent text. Decoder-only models like GPT use causal, left-to-right masking so a token sees only earlier tokens, and are pretrained on next-token prediction. This autoregressive objective makes them natural generators for chat, completion, and few-shot prompting. Encoder-decoder models like T5 combine both: a bidirectional encoder builds a representation of the input, and an autoregressive decoder with cross-attention generates the output. T5 is pretrained with span corruption, masking contiguous spans and reconstructing them, and frames every problem as text-to-text, making it strong for translation, summarization, and question answering where input and output are both sequences.

COMMON WRONG ANSWERS Saying BERT generates text autoregressively; calling GPT bidirectional; confusing span corruption with simple masked LM; ignoring cross-attention in encoder-decoder; assuming one architecture is best for everything.

LIKELY FOLLOW-UPS Why does causal masking enable generation? How does cross-attention work in T5? Why is BERT poor at generation? Could a decoder-only model do classification?

ONE CONCRETE EXAMPLE For extracting entities from contracts you fine-tune BERT, whose bidirectional context pinpoints span boundaries. For a customer-facing chatbot you use a GPT-style decoder that continues text token by token. For translating documents you use T5, whose encoder reads the source bidirectionally and decoder emits the target sequence.