Position-wise FFN: Each Token's Private Workshop

WHY IT EXISTS: A transformer cannot rely on attention alone. Attention is a weighted averaging operation across positions; it mixes context but is fundamentally linear and limited in how it can reshape features. The architecture needed a separate, non-linear processing step that could increase representational capacity, store factual associations, and act as a universal approximator within each layer without disturbing the sequence-wide information flow that attention provides.

THE MENTAL MODEL: Picture a factory assembly line. Attention is the robotic arm that passes each item to the correct neighboring stations so every part knows what the others are doing. The position-wise FFN is the dedicated toolhead that every item visits next: it stamps, drills, and polishes each item individually using the exact same machinery. No item learns about its neighbors during this step; it only receives the upgrade. In a transformer block, every token vector goes through this identical private workshop after the attention meeting.

HOW IT WORKS: The layer takes a single token vector and projects it into a higher-dimensional space, applies a non-linear activation, then projects it back to the original dimension. In the original Transformer this meant two linear layers with ReLU and a four-times expansion, so a 512-dimensional vector briefly became 2048. Modern architectures often use GELU or SwiGLU. The weights are shared across all positions in the sequence, but there is no interaction between positions. This means the FFN has no access to neighboring tokens; it trusts that attention has already folded the necessary context into the current vector before the FFN refines it.

WHEN TO USE IT: You encounter this in every standard transformer block, from BERT to GPT to T5. When designing or tuning a model, you adjust the FFN expansion ratio and activation function to trade off between capacity and compute. A larger hidden dimension gives the model more parameters and room to store knowledge, but it also increases memory bandwidth and inference cost.

WHEN NOT TO USE IT: Do not use or design an FFN if your goal is to model relationships between tokens in a single pass; that is the explicit job of the attention mechanism. Do not slash the FFN hidden size aggressively to make a model smaller, because the attention layers do not provide the non-linear feature expansion or the parameter density needed to maintain accuracy. Some modern architectures replace the dense FFN with a mixture-of-experts layer to keep capacity while sparsifying computation, but removing the position-wise non-linearity entirely breaks the transformer recipe.

ONE CANONICAL EXAMPLE: In the original Attention Is All You Need architecture with a model dimension of 512, each layer expands vectors to 2048 and projects them back. Because the FFN uses two weight matrices of size 512 by 2048 and 2048 by 512, it holds roughly twice as many parameters as the multi-head attention sublayer in the same block. This is why the FFN is often described as the memory warehouse of the transformer, while attention is merely the retrieval index.