Top 30 Transformers Interview Questions and Answers
30 multiple-choice questions on Transformers, drawn from 30 bites out of the 32 tagged Transformers on Tezvyn. Answer them here or read straight down. Every question carries the correct option, why it is correct, and a link to the bite it came from.
30 questions. Pick an answer, or open “Show the answer” to read it.
Answers are graded in your browser. Nothing is saved, and no XP or streak is earned here. The app keeps score.
Question 1 of 30
Why are positional encodings necessary before the first self-attention layer in a standard Transformer?
Show the answer
Answer: a · Because self-attention computes unordered pairwise dot products, leaving attention scores unchanged when tokens are permuted
Self-attention is permutation-invariant: swapping tokens does not change their dot-product attention scores, so the model receives no sequence-order signal without positional encodings. Distractor A represents the common misconception that feed-forward layers can learn implicit order, but they process each position independently and cannot recover shuffled order.
Read the full bite: Explain positional encodings in Transformers and their necessity
Question 2 of 30
Beyond modeling long-range dependencies, what key training advantage does self-attention have over an LSTM?
Show the answer
Answer: c · It can process all sequence positions in parallel rather than step by step
Self-attention computes all token interactions simultaneously, enabling parallel training that sequential LSTMs cannot match. It actually needs added positional encodings and is quadratic, not cheaper, in sequence length.
Read the full bite: Self-Attention versus Recurrent Architectures
Question 3 of 30
Why does self-attention use three separate learned projections of the same input rather than the raw embeddings directly?
Show the answer
Answer: a · It allows the model to learn which token features to use for matching versus which to pass forward as content
The correct answer reflects that learned projections decouple the matching process from content retrieval, letting the model decide which aspects of a token to use for scoring and which to propagate forward. The distractor describing a decoder query with encoder key and value defines cross-attention, not self-attention, where all three matrices are derived from the same input sequence.
Read the full bite: Explain Q, K, and V matrices in self-attention
Question 4 of 30
Why does multi-head attention generally not cost much more than a single full-width attention head?
Show the answer
Answer: c · The model dimension is split across heads, so each head operates in a smaller subspace
The total dimension is partitioned among heads, keeping aggregate compute comparable to one full head while gaining diverse attention patterns. Heads run in parallel with their own projections, not shared weights, and each still uses softmax.
Question 5 of 30
How does self-attention primarily determine the relevance of other words to a given word in a sequence?
Show the answer
Answer: d · By comparing the current word's Query vector with every other word's Key vector.
Self-attention calculates relevance by measuring the similarity between a word's Query vector and other words' Key vectors. This process allows it to directly identify and weigh the importance of all other words, unlike sequential processing or fixed context windows.
Read the full bite: Self-Attention: The Transformer's Core Idea
Question 6 of 30
During parallel teacher-forced training, what problem does the causal mask in decoder self-attention solve?
Show the answer
Answer: c · It stops position i from attending to future target tokens, preventing the model from copying answers
The causal mask forces token i to attend only to prior positions, preserving the autoregressive property and stopping the model from cheating by looking at future target tokens during parallel training. Option D describes the padding mask, which is a common misconception.
Read the full bite: What is masked in decoder self-attention and why?
Question 7 of 30
Which statement accurately captures the computational and memory complexity of exact self-attention as sequence length grows?
Show the answer
Answer: a · FLOPs grow as O(n^2) from the QK^T attention score matrix, while activation memory can be reduced to O(n) by recomputing rather than storing the full matrix.
Exact self-attention must perform O(n^2) FLOPs to compute the attention score matrix, but activation memory is not fundamentally quadratic: by recomputing scores instead of materializing the full n×n matrix in HBM, memory can be reduced to O(n). Option C is tempting because standard implementations often do store the full matrix, leading many to assume that O(n^2) memory is unavoidable even without approximation.
Read the full bite: Why is self-attention O(n^2) and what are the implications?
Question 8 of 30
Why is layer normalization preferred over batch normalization inside Transformer blocks for NLP?
Show the answer
Answer: c · Layer norm normalizes per token over features, independent of batch size and sequence length
Layer norm uses per-token feature statistics, so variable sequence lengths and small or padded batches do not destabilize it, and it matches at train and inference. Batch norm's batch-wise statistics are exactly what makes it unreliable here.
Read the full bite: Layer Norm and Residuals in Transformer Blocks
Question 9 of 30
What fundamental limitation necessitates the tokenization step for feeding text into a Transformer model?
Show the answer
Answer: b · Transformer models are mathematical functions that exclusively process numerical tensors.
The card explicitly states that "Neural networks, including Transformers, are mathematical functions that operate on numbers, not raw text strings." Thus, text must be converted into a numerical tensor format. While tokenization helps manage vocabulary and prepares for embeddings, the core reason is the model's numerical input requirement.
Read the full bite: Transformer Preprocessing: From Text to Tensors
Question 10 of 30
For which task would a Causal Language Model be the most suitable choice?
Show the answer
Answer: c · Extending a partial sentence into a complete, coherent paragraph
Causal Language Models are designed for open-ended text generation, predicting the next word based only on preceding words, making them ideal for continuing or extending text. Tasks like sentiment analysis, information extraction, or summarization typically require understanding the entire input, which is a limitation for CLMs.
Read the full bite: Causal Language Modeling: The Autocomplete Engine
Question 11 of 30
Which statement best describes how supervised fine-tuning updates a pre-trained language model?
Show the answer
Answer: b · It updates existing weights by predicting the next token only on curated assistant completions, masking the prompt.
Supervised fine-tuning updates existing weights by continuing next-token prediction on curated prompt-completion data, computing loss only on assistant tokens while masking prompts. The most tempting distractor is wrong because SFT deliberately uses curated demonstrations and a small learning rate, not raw internet text like pre-training.
Read the full bite: Describe supervised fine-tuning for a pre-trained language model
Question 12 of 30
Why does LoRA reduce GPU memory usage during fine-tuning compared to full fine-tuning?
Show the answer
Answer: b · It freezes the original weight matrix W and trains only low-rank matrices A and B, so gradients and optimizer states are only stored for A and B.
LoRA keeps the original weight matrix W frozen and only trains the small low-rank matrices A and B, which means gradients and optimizer states like those in Adam are only maintained for A and B, drastically reducing memory usage. The most tempting distractor claims the update is merged during training, but merging BA into W actually happens after training to preserve inference speed, not to save memory during fine-tuning.
Read the full bite: How does LoRA work and why is it memory-efficient?
Question 13 of 30
For which application scenario are adapter modules most beneficial for fine-tuning a large language model?
Show the answer
Answer: a · Adapting a single base model to numerous distinct downstream tasks with limited storage and compute.
Adapter modules are designed for efficiently adapting a single base model to multiple tasks, saving resources by only storing small, task-specific weights. While tempting, aiming for absolute maximum performance on a single, critical task might still favor full fine-tuning if resources are unlimited, as adapters might not match its performance in such cases.
Read the full bite: Adapter Modules: Efficient LLM Fine-Tuning
Question 14 of 30
What fundamental advantage does Transformer self-attention provide over RNNs for long sequences?
Show the answer
Answer: c · It allows parallel processing across tokens and constant-length paths between any two positions.
Self-attention creates direct connections between all positions in O(1) sequential steps and enables full parallelization during training, whereas RNNs require O(n) sequential unrolling with long gradient paths. Option B is tempting but wrong because the full attention matrix is actually quadratic in sequence length, not linear.
Read the full bite: Describe Transformer architecture and why self-attention beats recurrence
Question 15 of 30
When fine-tuning a pretrained model with the transformers library, why must the tokenizer be loaded from the same checkpoint as the model?
Show the answer
Answer: b · So the vocabulary, token IDs, and special tokens align with what the model was trained on
Each model expects a specific vocabulary and special-token scheme. A mismatched tokenizer produces token IDs the model never saw, corrupting inputs. Tokenizers are not interchangeable across models.
Read the full bite: Hugging Face Hub, transformers, and datasets
Question 16 of 30
Which pretraining objective and masking scheme makes decoder-only models like GPT naturally suited to text generation?
Show the answer
Answer: d · Causal left-to-right masking with next-token prediction
Decoder-only models mask future tokens and learn next-token prediction, exactly the autoregressive setup generation needs. Bidirectional masked LM describes BERT, and span corruption with cross-attention describes T5.
Read the full bite: Encoder, decoder, and encoder-decoder Transformers
Question 17 of 30
What is the main reason to use multi-head attention instead of a single attention mechanism?
Show the answer
Answer: c · To enable the model to focus on various aspects and relationships within the data concurrently.
Multi-head attention's primary benefit is allowing the model to analyze different types of relationships (like syntax and semantics) simultaneously, providing a more comprehensive understanding. Option B is incorrect because the card states that multi-head attention adds significant computational overhead.
Read the full bite: Multi-Head Attention: Seeing Data From Multiple Angles
Question 18 of 30
What core mechanism lets Transformers capture long-range dependencies more effectively than recurrent networks?
Show the answer
Answer: c · Self-attention, where every token directly attends to every other token
Self-attention connects any two tokens directly in a single step, regardless of distance, enabling parallel global context. Recurrence (option B) is exactly what the Transformer replaced because long-range signals degrade across many steps.
Question 19 of 30
Which scenario most appropriately calls for the use of cross-attention?
Show the answer
Answer: c · Aligning spoken words with corresponding visual actions in a video.
Cross-attention is designed to fuse information from fundamentally different sources, such as aligning audio (spoken words) with visual frames in a video. Options A, B, and D describe tasks within a single modality where self-attention would be the appropriate mechanism for understanding internal relationships or processing.
Read the full bite: Cross-Attention: How Models Fuse Text and Images
Question 20 of 30
What is the fundamental mechanism FlashAttention employs to achieve its performance and memory efficiency gains?
Show the answer
Answer: c · It minimizes data transfers to and from slow High Bandwidth Memory (HBM) by processing blocks in fast on-chip SRAM.
FlashAttention's core innovation is its IO-awareness, which minimizes slow data transfers between GPU's main memory (HBM) and faster on-chip memory (SRAM) by processing attention in blocks within SRAM. Option A is incorrect because FlashAttention computes the exact same output as standard attention, implying similar FLOPs, but with restructured execution.
Read the full bite: FlashAttention: Faster, Memory-Efficient Exact Attention
Question 21 of 30
In cross-attention for VQA, where do the queries, keys, and values come from?
Show the answer
Answer: a · Queries from one modality, keys and values from the other
Cross-attention uses queries from one modality and keys and values from the other, enabling text tokens to attend over visual features. Drawing all three from one modality describes self-attention instead.
Read the full bite: Cross-attention in transformer VQA models
Question 22 of 30
Which architectural aspect primarily contributes to the computational efficiency of Masked Autoencoders during pre-training?
Show the answer
Answer: c · The encoder processes only the visible, unmasked image patches.
Option C is correct because the card states that feeding only the visible patches into the encoder is a 'crucial design choice' that 'makes the process highly efficient' as the encoder, the most computationally expensive part, processes only a small fraction of the input. While the decoder is lightweight (Option A), the primary efficiency gain for the overall process stems from the encoder's reduced input.
Read the full bite: Masked Autoencoders: Learning Vision by Filling in the Blanks
Question 23 of 30
In a Transformer encoder-decoder, what is the specific role of the decoder's cross-attention sublayer?
Show the answer
Answer: a · It lets each decoder position attend to the encoder's output representations of the input
Cross-attention is where decoder queries attend to the encoder's keys and values, conditioning generation on the source. Blocking future tokens is masked self-attention's job, and bidirectional input attention happens in the encoder.
Read the full bite: Transformer Encoder-Decoder Architecture
Question 24 of 30
When fine-tuning a pretrained LLM, why is it generally unsafe to replace its tokenizer with a new one?
Show the answer
Answer: a · The model's knowledge is coupled to specific token IDs, so new IDs would retrieve unrelated learned vectors
The card states that a pretrained model's knowledge is tightly coupled to its original token IDs, so a new tokenizer maps text to different IDs and retrieves the wrong embedding vectors, producing garbage output. Distractor A is wrong because linear projections are only for inherently continuous inputs like raw audio or pixels, whereas text always uses an embedding lookup table.
Read the full bite: Tokenization and Input Embeddings in LLMs
Question 25 of 30
Your intern suggests removing the position-wise FFN to speed up inference, arguing that attention already mixes token information. What critical capability would the model lose?
Show the answer
Answer: d · The ability to apply non-linear, per-token feature expansion and store factual associations
Attention mixes context across positions but is fundamentally linear and cannot reshape features or store factual associations on its own; the FFN provides that private, non-linear per-token workshop. Option C is tempting but wrong because communicating between positions is attention's explicit job, whereas the FFN never shares between seats.
Read the full bite: Position-wise FFN: Each Token's Private Workshop
Question 26 of 30
A team uses a pure stack of Transformer encoder blocks for autoregressive text generation. What is the fundamental flaw in this approach?
Show the answer
Answer: d · The encoder's bidirectional self-attention lets tokens attend to future positions, violating causal generation constraints
The card identifies the encoder block's bidirectional self-attention as the core problem for generation because it allows tokens to see future information, violating causality. The most tempting distractor incorrectly claims the encoder is sequential like an RNN, but the card emphasizes that encoder blocks perform parallel attention across the entire sequence to eliminate that very bottleneck.
Question 27 of 30
In a Transformer for machine translation, where should masked multi-head attention be used to ensure correct autoregressive behavior?
Show the answer
Answer: a · In the decoder's self-attention to let each output position attend only to prior positions
Masked multi-head attention belongs in decoder self-attention to block future target tokens during training, matching left-to-right inference. Applying it in the encoder (D) destroys bidirectional context, while cross-attention (B) should see the full encoder output.
Question 28 of 30
Why does masked language modeling struggle to produce coherent multi-sentence paragraphs?
Show the answer
Answer: c · It learns to reconstruct scattered holes rather than generate left-to-right
MLM is trained to fill random blanks using bidirectional context, so it never learns the causal left-to-right structure needed for sequential generation. Distractor A is wrong because MLM explicitly attends to both directions—it is unidirectional models that only see previous tokens.
Read the full bite: Masked Language Modeling: Fill-in-the-Blank Pretraining
Question 29 of 30
What is the primary architectural advantage of self-attention over traditional Recurrent Neural Networks (RNNs) for processing sequences?
Show the answer
Answer: c · It allows for parallel computation by processing all sequence elements simultaneously.
Self-attention was developed to overcome the sequential processing bottleneck of RNNs, enabling all elements in a sequence to be processed in parallel, which drastically speeds up training. Option D is incorrect because self-attention's core function is to weigh the importance of different words, not to treat them equally.
Read the full bite: Self-Attention: How Models Weigh Word Importance
Question 30 of 30
When considering an autoregressive model for image generation, what is the most significant trade-off users must accept?
Show the answer
Answer: c · Achieving maximum image quality and logical coherence versus the inherent slowness of the generation process.
The card explicitly states that autoregressive models create high-fidelity, coherent images but are much slower due to their sequential nature, highlighting this as the key trade-off. While other factors like training efficiency (Option B) or architectural complexity (Option A) are relevant, they are not presented as the primary trade-off for *using* these models.
Read the full bite: Autoregressive Models: Generating Images One 'Word' at a Time
Could you explain these out loud?
That is what an interview actually tests. Tezvyn gives you questions like these with what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.