Transformer Encoder Block

WHY IT EXISTS: Before transformers, sequence models processed tokens one at a time. RNNs and LSTMs were slow to train and struggled to connect distant tokens because signals had to propagate through every intermediate step. The encoder block was invented to let every position in a sequence directly attend to every other position in a single parallel pass, removing the sequential bottleneck and making long-range dependencies as cheap as local ones.

THE MENTAL MODEL: Imagine a room full of people all updating their notes at the same time. Each person looks at everyone else in the room, decides who is relevant to their current topic, and blends those people's notes into their own. After that blending, each person privately thinks through a more complex transformation of their updated note. That is the encoder block: a public context-mixing step followed by a private processing step, repeated across many layers.

HOW IT WORKS: The block receives a matrix of input vectors that already include positional information. It creates three copies of this matrix called queries, keys, and values. For every token, its query vector is compared against all key vectors to produce attention scores. These scores are scaled by the square root of the head dimension and passed through a softmax to yield a probability distribution. Each token then becomes the weighted sum of all value vectors according to that distribution. The results are concatenated from multiple heads, projected linearly, and passed through a feed-forward network that typically expands the hidden dimension by four, applies a nonlinearity, and projects back down. Both the attention and feed-forward sublayers have residual connections around them, and layer normalization is applied to keep activations stable during training.

WHEN TO USE IT: Use an encoder block when you need a deep bidirectional representation of an entire input sequence and the full text is available before inference begins. This is the standard choice for text classification, named entity recognition, semantic search, and sentence embedding models where left and right context must be fused into a single vector. It is also useful in encoder-decoder architectures like the original Transformer for machine translation, where the encoder first builds a rich source representation for the decoder to query.

WHEN NOT TO USE IT: Do not use a pure encoder block for autoregressive language modeling or text generation. Because self-attention is bidirectional, each token can see future tokens, which violates the causal structure of generation. Decoder-only models fix this by masking the attention matrix so positions can only attend to themselves and previous tokens. Encoder blocks are also inefficient for extremely long sequences when quadratic attention cost becomes prohibitive, and they are unnecessary for simple tasks where bag-of-words or convolutional features are sufficient.

ONE CANONICAL EXAMPLE: BERT is built entirely from a stack of twelve or more identical Transformer encoder blocks. During pretraining, each block allows every word in a sentence to attend to every other word, enabling the model to learn deep bidirectional context. When you feed BERT the sentence "The cat sat on the mat," the encoder blocks compute contextualized embeddings where the representation of "sat" is directly influenced by both "cat" and "mat" simultaneously. These rich vectors are then passed to a classification head or used directly for semantic similarity search.