Patch Embedding: Turning Images into Words for Transformers

WHY IT EXISTS Transformers were designed for sequential data like text, not 2D grids like images. Processing an image pixel-by-pixel would create an input sequence too long for a Transformer to handle efficiently. Patch embedding was created to convert images into a manageable, sequence-like format that these powerful models can understand.

THE MENTAL MODEL Think of an image as a mosaic. Patch embedding is like breaking that mosaic apart into its individual tiles. You then describe each tile with a set of numbers (a vector) and line them up in a row. This row of tile descriptions is then "read" by the Transformer, which learns how the tiles relate to each other to form the bigger picture, just as it learns how words relate in a sentence.

HOW IT WORKS An input image is first divided into a grid of fixed-size, non-overlapping patches. For example, a 224x224 pixel image might be split into a 14x14 grid of 16x16 pixel patches. Each 2D patch is then "flattened" or serialized into a single long vector. A 16x16 RGB patch becomes a vector of 16 * 16 * 3 = 768 numbers. Finally, this long vector is multiplied by a trainable weight matrix to project it into a smaller, consistent dimension (the embedding). This final vector is the patch embedding.

WHEN TO USE IT This technique is the defining first step for any Vision Transformer (ViT) architecture. Use it whenever you want to apply a standard Transformer model to a computer vision task, such as image classification or object detection, without relying on traditional convolutional layers as the primary feature extractor.

WHEN NOT TO USE IT Avoid this approach if your task depends critically on fine-grained, pixel-perfect spatial information from the very start. The patching process inherently loses the precise spatial relationships within each patch. For tasks where this local detail is paramount and computational cost is a major factor, a classic Convolutional Neural Network (CNN) may be more suitable.

ONE CANONICAL EXAMPLE The original Vision Transformer model takes a 224x224 pixel image and splits it into 196 patches, each 16x16 pixels. Each patch is flattened into a 768-dimensional vector (16 * 16 * 3). This vector is then linearly projected to the model's hidden dimension, creating a sequence of 196 embeddings ready for the Transformer encoder.