Transformers
32 bites tagged Transformers — interview questions with model answers, and 60-second explainers.
Self-Attention: The Transformer's Core Idea
Self-attention lets a model weigh the importance of different words in a sequence to understand context. This core mechanism of the transformer architecture powers LLMs for translation and generation.
Autoregressive Models: Generating Images One 'Word' at a Time
Autoregressive models generate images sequentially, like writing a sentence word by word. Each new pixel or patch depends on what came before. This creates high-fidelity images but is much slower than one-shot models, a key trade-off in generative AI.
Self-Attention: How Models Weigh Word Importance
Self-attention lets a model weigh the importance of all words in a sequence simultaneously, asking "which other words are most relevant?" It's the core of Transformers, enabling parallel processing for tasks like translation, unlike sequential RNNs.
Masked Autoencoders: Learning Vision by Filling in the Blanks
Masked Autoencoders (MAEs) teach models vision by playing "fill-in-the-blanks" with images, masking most of an image (e.g., 75%) and learning to reconstruct it. This is used for self-supervised pre-training of large Vision Transformers on unlabeled data.
Cross-Attention: How Models Fuse Text and Images
Cross-attention lets a model fuse different data streams, like asking 'what in this image corresponds to this word?'. It's key for text-to-image generation, where text queries attend to image features. The footgun is confusing it with self-attention.
Multi-Head Attention: Seeing Data From Multiple Angles
Multi-head attention lets a model analyze a sequence from multiple perspectives at once. It runs several "attention heads" in parallel, each focusing on different relationships, like syntax vs. semantics.
Encoder-Only vs. Decoder-Only vs. Encoder-Decoder Transformers?
This tests your ability to connect transformer architecture to specific NLP tasks. A great answer explains how each model's attention mechanism dictates its use: encoder-only (bidirectional attention) for understanding content, decoder-only (causal attention) for text generation, and encoder-decoder for sequence-to-sequence tasks like translation. The key red flag is failing to explain the *why* behind the task suitability—the attention mechanism.
What is the vanishing gradient problem and how do transformers avoid it?
This tests your understanding of core deep learning training issues and the transformer's specific architectural solutions. A great answer defines vanishing gradients in sequential models, then explains how the transformer's parallel attention mechanism creates direct, short paths for gradients between any two tokens, regardless of distance. A red flag is vaguely mentioning 'attention' without explaining why its parallel nature is the key to solving the problem for long sequences.
Get Transformers bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.