Explain the concept of self-attention

What this tests: Your understanding of the fundamental mechanism that allows Transformer models to process sequences by relating different positions to each other. It's a check for core competency in modern deep learning architectures, moving beyond a surface-level definition.

A good answer covers: Purpose: Self-attention allows a model to weigh the importance of different words in the same sequence* to update the representation of each word. This is how the model builds contextual understanding. • QKV Vectors: For each input token, the model learns to project its embedding into three separate vectors: a Query (Q), a Key (K), and a Value (V). The Query represents the current token's need for information, the Key represents what information a token has, and the Value is the content to be passed along. • Score Calculation: The attention score is calculated by taking the dot product of a token's Query vector with the Key vectors of all other tokens in the sequence (including itself). This measures the compatibility or relevance between tokens. • Weighting: These raw scores are scaled (typically by the square root of the key dimension) and then passed through a softmax function. This converts the scores into positive weights that sum to 1, representing the "attention" distribution for the querying token. • Final Output: The output representation for the token is a weighted sum of all the Value vectors in the sequence, using the softmax-normalized attention weights. This new representation is context-aware.

Common wrong answers: • Vague Definition: Only stating that "it helps the model pay attention to important words" without explaining the Q, K, V mechanism that makes it possible. • Confusing with other attention: Describing older sequence-to-sequence attention mechanisms (like in RNNs) which operate between an encoder and a decoder, rather than within a single sequence. Forgetting the "self" part: Failing to mention that the Queries, Keys, and Values all originate from the same* input sequence.

Follow-up the interviewer might ask: • "How is this different from multi-head attention?" • "What is the computational complexity of this layer with respect to sequence length N, and why is that a problem?"

One concrete example: In the sentence "The robot picked up the ball because it was heavy", when processing the word "it", self-attention helps determine what "it" refers to. The Query vector for "it" would have a high dot-product similarity with the Key vector for "ball", not "robot". This results in a high attention weight between "it" and "ball". The final representation for "it" will then be heavily influenced by the Value vector of "ball", effectively embedding the pronoun's antecedent into its representation.