Transformers: Processing Language in Parallel with Attention
Transformers process all input text at once, weighing which words are most important to each other in parallel. This 'attention' mechanism is the core of models like GPT, allowing them to understand context over long sequences. Text is broken into tokens, turned into vectors, and then contextualized by multiple attention 'heads'. The key footgun is that attention alone is order-agnostic; without explicit positional encodings, the model can't distinguish 'dog bites man' from 'man bites dog'.
### The Mental Model
Think of a Transformer not as reading a sentence word-by-word, but as looking at the entire text at once and drawing lines of connection between related words. The stronger the connection, the more 'attention' one word pays to another. This happens in parallel for all words, allowing the model to build a rich, contextual understanding of the entire input simultaneously.
### How It Works
Input text is first broken down into numerical representations called tokens. Each token is then converted into a high-dimensional vector using a lookup embedding table. At each layer, the model uses a multi-head attention mechanism to contextualize every token. For each token, the model calculates a Query, Key, and Value vector. The Query from one token is compared against the Keys of all other tokens to produce attention scores. These scores determine how much of each token's Value should contribute to the current token's new representation. This process is done multiple times in parallel ('multi-head'), allowing the model to capture different types of relationships simultaneously.
### When to Use It
* **Natural Language Processing:** The primary use case, forming the basis of LLMs like GPT and BERT for tasks like translation, summarization, and generation. * **General Sequence Modeling:** The architecture is effective for any sequential data, including time-series analysis, music generation, and even computer vision (by treating image patches as a sequence).
### When NOT to Use It
* **When Order is Irrelevant:** If your task is simple document classification based on word counts (a 'bag-of-words' approach), the complexity and computational cost of a Transformer is likely overkill. * **Without Positional Information:** A raw self-attention mechanism is permutation-invariant. Using it without adding positional encodings is a critical error that scrambles the meaning of ordered sequences. The model would see "A then B" and "B then A" as identical.
### One Canonical Example
Consider the sentence: "The tired developer wrote code". When the model processes the word "wrote", the attention mechanism might assign high scores to "developer" (who wrote?) and "code" (what was written?), while giving a lower score to "The". Multi-head attention allows it to do this in several ways at once; one 'head' might focus on the subject-verb relationship ("developer" → "wrote"), while another focuses on the verb-object relationship ("wrote" → "code").
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.