Transformer Architecture

Why it exists

Recurrent networks process sequences one step at a time, which is slow to train and struggles to carry information across long distances because signals must pass through many steps. The Transformer was introduced to remove recurrence entirely, enabling parallel computation over a sequence and direct connections between distant tokens, which solved both the speed and long-range dependency problems.

The mental model

Imagine each word in a sentence asking every other word how relevant it is, then building its own meaning as a weighted blend of the most relevant words. That weighted lookup is self-attention. Because every word can consult every other word in one operation, context is gathered globally and simultaneously rather than passed down a chain.

How it works

Each token is projected into query, key, and value vectors. Attention scores come from comparing a token's query with every token's key; those scores are normalized and used to take a weighted sum of value vectors, producing a context-aware representation. Multi-head attention runs several such attention computations in parallel to capture different relationships. Since attention is order-agnostic, positional encodings are added so the model knows token order. Each layer combines attention with a position-wise feed-forward network, residual connections, and normalization, and many layers are stacked. Variants are encoder-only like BERT, decoder-only like GPT, or encoder-decoder for translation.

When it matters

Transformers dominate natural language processing and increasingly vision, audio, and multimodal tasks. They matter most when you have large data and compute and need to model long-range structure. The main cost to know is that standard self-attention scales quadratically with sequence length, which motivates efficient-attention variants for very long inputs.

A concrete example

Consider the sentence the animal did not cross the street because it was too tired. To resolve what it refers to, a Transformer's attention lets the token it place high weight on animal rather than street, directly linking the two words despite the distance between them. This single global attention step captures the dependency that a recurrent model would have to carry across many sequential steps, illustrating why Transformers handle long-range context so effectively.