Autoregressive Models: Generating Images One 'Word' at a Time
Autoregressive models generate images sequentially, like writing a sentence word by word. Each new pixel or patch depends on what came before. This creates high-fidelity images but is much slower than one-shot models, a key trade-off in generative AI.
WHY IT EXISTS Autoregressive models were invented to apply the proven power of sequential modeling from natural language processing to the visual domain. While images are spatial, not inherently sequential, treating them as a sequence allows a model to generate complex, coherent scenes in a principled, step-by-step fashion.
THE MENTAL MODEL Think of an autoregressive model as an artist that paints an image pixel by pixel, or patch by patch, in a fixed order (e.g., top-left to bottom-right). At each step, it looks at every pixel it has already painted to decide the exact color of the very next one. It repeats this process thousands of times until the entire canvas is filled.
HOW IT WORKS An autoregressive model factorizes the joint probability distribution of an image's pixels (or tokens) into a product of conditional probabilities. To generate an image, it first defines a strict ordering of the visual elements. Then it samples the first element, feeds it back as input to sample the second, feeds the first two back to sample the third, and so on. This process can happen at different levels of abstraction. Three common strategies are: pixel-based (predicting one pixel at a time), token-based (predicting discrete patches or 'visual words'), and scale-based (generating a low-resolution image then autoregressively adding detail).
WHEN TO USE IT Use autoregressive models when maximum image quality and logical coherence are the top priorities, and generation speed is a secondary concern. They excel at capturing long-range dependencies, making them suitable for generating complex textures and detailed scenes. Their application extends beyond static images to video, 3D models, and multimodal generation where sequence is more natural.
WHEN NOT TO USE IT Avoid autoregressive models for any application requiring fast or real-time generation. The sequential, element-by-element nature is a fundamental bottleneck that cannot be easily parallelized during inference. This makes them significantly slower than other generative architectures like GANs or diffusion models.
ONE CANONICAL EXAMPLE A token-based autoregressive image generator first uses an encoder, like a VQ-VAE, to compress an image into a smaller grid of discrete 'visual tokens' from a learned codebook. Then, a large transformer model, similar to one used for text, is trained to predict the next token in the flattened sequence. During inference, the transformer generates a full sequence of tokens, which are then passed to a decoder to reconstruct a high-resolution image.
Read the original → arxiv.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.