Transformers
32 bites tagged Transformers: interview questions with model answers, and 60-second explainers.
Test yourself: Top 30 Transformers interview questions →Multiple choice, with the correct answer and why it is correct on every question. Free, no sign-in.Encoder, decoder, and encoder-decoder Transformers
Encoder-only uses bidirectional masked-token pretraining for understanding tasks, decoder-only uses causal next-token prediction for generation, encoder-decoder uses span corruption for…
Hugging Face Hub, transformers, and datasets
The Hub hosts models and data, transformers loads models and tokenizers and provides the Trainer, datasets streams and maps preprocessing.
Transformer Architecture
The Transformer replaces recurrence with self-attention, letting every token directly attend to every other token in parallel. This enables long-range context and fast training on GPUs, making it the backbone of modern large language models and much of…
Cross-attention in transformer VQA models
Text queries attend over image regions, learning alignment that grounds words to visual content.
Layer Norm and Residuals in Transformer Blocks
Residuals preserve gradient flow, layer norm stabilizes activations per token, and it beats batch norm because it is independent of batch and sequence length.
Why Multi-Head Attention
Multiple heads attend to different subspaces and relations in parallel, which one big head averages away.
Self-Attention versus Recurrent Architectures
Each token attends to all others via query-key-value, enabling parallelism and direct long-range links.
Transformer Encoder-Decoder Architecture
The encoder-decoder Transformer maps an input sequence into rich contextual representations with an encoder, then a decoder generates output tokens autoregressively while attending to those representations via cross-attention, making it ideal for…
How does LoRA work and why is it memory-efficient?
LoRA freezes weights and trains A and B so delta-W equals BA, cutting trainable params 10,000x and memory 3x since only A and B get grads.
Describe supervised fine-tuning for a pre-trained language model
Tests if you know SFT aligns a base model to instructions using curated prompt-completion data. A strong answer covers next-token prediction on completions, conversational formats, and small learning rates.
Why is self-attention O(n^2) and what are the implications?
Tests the attention matrix bottleneck. Strong answers note QK^T yields an N×N matrix, creating quadratic compute and memory that blocks long documents and high-res images. Red flag: confusing model size with activation memory.
What is masked in decoder self-attention and why?
Future positions are masked so token i attends only to prior tokens. This prevents cheating during parallel teacher-forced training when the full target is visible.
Explain Q, K, and V matrices in self-attention
This tests the information-retrieval intuition behind self-attention. Cover that Q, K, V are linear projections of one input; Q requests, K indexes, V supplies content; scores weight a sum of V.
Explain positional encodings in Transformers and their necessity
Explain encodings inject order into embeddings; cite sinusoidal or learned vectors.
Masked Language Modeling: Fill-in-the-Blank Pretraining
MLM hides random tokens and trains the model to reconstruct them from context. It powers BERT-style encoders for classification and search. The catch is that it never learns left-to-right generation, so it needs extra machinery for open-ended text.
Masked Multi-Head Attention in Decoders
Masked multi-head attention runs parallel detectors over past tokens only, stopping a decoder from peeking ahead. It powers autoregressive models like GPT. The footgun is using the causal mask in bidirectional encoders, which silently destroys context.
Transformer Encoder Block
A Transformer encoder block mixes full sequence context in parallel: every token attends to all others to refine its vector. It drives bidirectional models like BERT. The footgun is using it unmasked for generation, which leaks future information.
Position-wise FFN: Each Token's Private Workshop
Think of the position-wise FFN as each token's private gym after attention: it bulks up features and stores facts, but never shares between seats. Cutting it to save parameters starves the model because attention cannot do this alone.
Tokenization and Input Embeddings in LLMs
Tokenization splits language into tokens, and embeddings map token IDs into vectors with meaning. Every transformer does this first. The footgun is assuming one token equals one word—token counts behave unpredictably when words merge or split.
Describe Transformer architecture and why self-attention beats recurrence
This tests parallelization and long-range dependencies. A strong answer outlines the encoder-decoder stack with multi-head self-attention, contrasts O(1) sequential steps versus RNNs' O(n) unrolling, and warns that describing it as averaging misses key ideas.
FlashAttention: Faster, Memory-Efficient Exact Attention
FlashAttention is an IO-aware algorithm that computes exact attention faster and with less memory. It avoids slow GPU memory transfers, making it a key optimization for training and serving large models on modern GPUs.
Adapter Modules: Efficient LLM Fine-Tuning
Adapters are small modules plugged into a frozen LLM to avoid costly full fine-tuning. This lets you specialize a base model for many tasks by training tiny, swappable plugins instead of duplicating the entire model for each task.
Causal Language Modeling: The Autocomplete Engine
During each prediction, causal attention uses only the left context; a prompt can still contain a complete document, and the model can generate a continuation or answer from that prompt.
Transformer Preprocessing: From Text to Tensors
Transformers don't read text; they read numbers. A tokenizer is the translator, converting sentences into numerical tensors the model understands. This is the mandatory first step for any NLP task. The footgun is using a tokenizer that doesn't match the model.
Get Transformers bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.