Transformers
32 bites tagged Transformers — interview questions with model answers, and 60-second explainers.
Encoder, decoder, and encoder-decoder Transformers
Encoder-only uses bidirectional masked-token pretraining for understanding tasks, decoder-only uses causal next-token prediction for generation, encoder-decoder uses span corruption for… mapping architecture to objective and use case.
Hugging Face Hub, transformers, and datasets
The Hub hosts models and data, transformers loads models and tokenizers and provides the Trainer, datasets streams and maps preprocessing. practical fluency with the standard NLP toolchain.
Transformer Architecture
The Transformer replaces recurrence with self-attention, letting every token directly attend to every other token in parallel. This enables long-range context and fast training on GPUs, making it the backbone of modern large language models and much of…
Cross-attention in transformer VQA models
Text queries attend over image regions, learning alignment that grounds words to visual content. vision-language fusion via attention. confusing self-attention with cross-attention or saying it just concatenates features.
Layer Norm and Residuals in Transformer Blocks
Residuals preserve gradient flow, layer norm stabilizes activations per token, and it beats batch norm because it is independent of batch and sequence length. how Transformer blocks stay trainable at depth.
Why Multi-Head Attention
Multiple heads attend to different subspaces and relations in parallel, which one big head averages away. rationale for splitting attention into heads. claiming more heads is always better or that it raises total compute.
Self-Attention versus Recurrent Architectures
Each token attends to all others via query-key-value, enabling parallelism and direct long-range links. understanding self-attention and its edge over RNNs.
Transformer Encoder-Decoder Architecture
The encoder-decoder Transformer maps an input sequence into rich contextual representations with an encoder, then a decoder generates output tokens autoregressively while attending to those representations via cross-attention, making it ideal for…
How does LoRA work and why is it memory-efficient?
LoRA freezes weights and trains A and B so delta-W equals BA, cutting trainable params 10,000x and memory 3x since only A and B get grads. Low-rank adaptation. Claiming it shrinks size or adds latency.
Describe supervised fine-tuning for a pre-trained language model
Tests if you know SFT aligns a base model to instructions using curated prompt-completion data. A strong answer covers next-token prediction on completions, conversational formats, and small learning rates.
Why is self-attention O(n^2) and what are the implications?
Tests the attention matrix bottleneck. Strong answers note QK^T yields an N×N matrix, creating quadratic compute and memory that blocks long documents and high-res images. Red flag: confusing model size with activation memory.
What is masked in decoder self-attention and why?
Future positions are masked so token i attends only to prior tokens. This prevents cheating during parallel teacher-forced training when the full target is visible. Causal constraints in decoder training.
Explain Q, K, and V matrices in self-attention
This tests the information-retrieval intuition behind self-attention. Cover that Q, K, V are linear projections of one input; Q requests, K indexes, V supplies content; scores weight a sum of V.
Explain positional encodings in Transformers and their necessity
Explain encodings inject order into embeddings; cite sinusoidal or learned vectors. Self-attention is permutation-invariant, requiring position signals. Claiming attention learns token order without position info.
Masked Language Modeling: Fill-in-the-Blank Pretraining
MLM hides random tokens and trains the model to reconstruct them from context. It powers BERT-style encoders for classification and search. The catch is that it never learns left-to-right generation, so it needs extra machinery for open-ended text.
Masked Multi-Head Attention in Decoders
Masked multi-head attention runs parallel detectors over past tokens only, stopping a decoder from peeking ahead. It powers autoregressive models like GPT. The footgun is using the causal mask in bidirectional encoders, which silently destroys context.
Transformer Encoder Block
A Transformer encoder block mixes full sequence context in parallel: every token attends to all others to refine its vector. It drives bidirectional models like BERT. The footgun is using it unmasked for generation, which leaks future information.
Position-wise FFN: Each Token's Private Workshop
Think of the position-wise FFN as each token's private gym after attention: it bulks up features and stores facts, but never shares between seats. Cutting it to save parameters starves the model because attention cannot do this alone.
Tokenization and Input Embeddings in LLMs
Tokenization splits language into tokens, and embeddings map token IDs into vectors with meaning. Every transformer does this first. The footgun is assuming one token equals one word—token counts behave unpredictably when words merge or split.
Describe Transformer architecture and why self-attention beats recurrence
This tests parallelization and long-range dependencies. A strong answer outlines the encoder-decoder stack with multi-head self-attention, contrasts O(1) sequential steps versus RNNs' O(n) unrolling, and warns that describing it as averaging misses key ideas.
FlashAttention: Faster, Memory-Efficient Exact Attention
FlashAttention is an IO-aware algorithm that computes exact attention faster and with less memory. It avoids slow GPU memory transfers, making it a key optimization for training and serving large models on modern GPUs.
Adapter Modules: Efficient LLM Fine-Tuning
Adapters are small modules plugged into a frozen LLM to avoid costly full fine-tuning. This lets you specialize a base model for many tasks by training tiny, swappable plugins instead of duplicating the entire model for each task.
Causal Language Modeling: The Autocomplete Engine
Causal Language Modeling is like a powerful autocomplete, predicting the next word based only on what came before. It's the engine for text generation in chatbots, creative writing tools, and coding assistants. The footgun: it can't see future words.
Transformer Preprocessing: From Text to Tensors
Transformers don't read text; they read numbers. A tokenizer is the translator, converting sentences into numerical tensors the model understands. This is the mandatory first step for any NLP task. The footgun is using a tokenizer that doesn't match the model.
Get Transformers bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.