Causal versus Masked Language Modeling

WHAT THIS TESTS This checks foundational knowledge of how large language models acquire general capabilities before any task-specific tuning, and the key architectural-objective distinction between GPT-style and BERT-style models.

A GOOD ANSWER COVERS The goal of pre-training is to learn broad, transferable representations of language and world knowledge from enormous amounts of unlabeled text using self-supervision, where the labels come from the text itself. Causal language modeling, the GPT objective, trains the model to predict the next token given only the tokens before it, using a causal mask so each position cannot see the future. Because generation is exactly next-token prediction, this objective is inherently suited to text generation. Masked language modeling, the BERT objective, randomly masks a portion of input tokens and trains the model to reconstruct them using context from both directions. This bidirectional view produces rich representations well-suited to understanding tasks like classification and extraction, but the model is not a natural left-to-right generator.

COMMON WRONG ANSWERS Claiming BERT generates text autoregressively; its bidirectional objective does not lend itself to free generation. Saying GPT uses bidirectional context; the causal mask forbids it. Conflating the pre-training objective with downstream fine-tuning. Forgetting that MLM only computes loss on the masked positions.

LIKELY FOLLOW-UPS Why does bidirectional context help understanding but hurt generation? What is the pretrain-finetune discrepancy from the mask token, and how do variants address it? How do sequence-to-sequence objectives like T5's span corruption combine both ideas? Why have decoder-only causal models come to dominate large-scale generative AI?

ONE CONCRETE EXAMPLE Given the sentence about a cat sitting on a mat, a causal model is trained so that after seeing the cat sat on the it must predict mat, seeing only the left context. A masked model instead sees the cat sat on the [MASK] with the full surrounding sentence visible and must recover mat using both the words before and any words after, which is why it builds deeper contextual understanding but is not used to generate text token by token.