What is masked in decoder self-attention and why?

WHAT THIS TESTS: Your understanding of causal masking in the decoder self-attention layer and its role in preserving the autoregressive property during parallel training. Interviewers want to see that you distinguish bidirectional encoder attention from unidirectional decoder attention, and that you know why teacher forcing requires an explicit lookahead mask.

A GOOD ANSWER COVERS: Four things in order. First, what is masked: future positions in the target sequence. When computing self-attention inside the decoder, attention scores for positions j greater than i are set to negative infinity before the softmax step, causing them to receive zero weight after softmax. Second, the mechanism: this is often called a lookahead mask or causal mask, implemented via a lower-triangular matrix. Third, why it is critical: during training, the decoder receives the full correct target sequence shifted right by one position and processes all positions in parallel. Without masking, the query at position i could attend to target tokens at position i or beyond, letting the model cheat by looking at the answer it must predict. Fourth, inference alignment: at inference time the model generates tokens one by one, naturally only accessing prior tokens. Masking during training enforces this same causal constraint so the training distribution matches the inference distribution.

COMMON WRONG ANSWERS: Three red flags. First, confusing the causal mask with padding masks. Padding masks hide pad tokens across the batch; the decoder self-attention mask prevents looking at future tokens. Second, claiming the encoder uses masked self-attention. The encoder uses unmasked bidirectional self-attention so every token can see every other token. Third, saying the mask hides the source sequence from the decoder. The source sequence is not visible to decoder self-attention by architecture; cross-attention later allows the decoder to attend to encoder outputs.

LIKELY FOLLOW-UPS: The interviewer may ask how the mask is implemented numerically, expecting you to mention adding negative infinity to the upper triangle of the attention score matrix before softmax. They may ask what happens if you remove the mask during training, expecting you to explain that the model would learn to copy future tokens and fail at autoregressive generation. Another follow-up is comparing this to bidirectional attention in BERT-style encoders.

ONE CONCRETE EXAMPLE: Imagine translating a sentence where the target sequence is "I love cats". During training, all three tokens are fed into the decoder simultaneously. When the decoder processes the token at position 2 representing "love", the causal mask ensures it can only attend to position 1 ("I") and itself at position 2, but not position 3 ("cats"). If position 3 were visible, the model could trivially predict "cats" by copying it from the future, destroying the autoregressive learning signal.