Diffusion Models: Generating by Reversing Noise

Why it exists

Generative models like GANs were often unstable to train and could suffer from 'mode collapse,' where they only produce a limited variety of outputs. Diffusion models were developed to provide a more stable training objective that leads to higher-quality and more diverse results, at the cost of slower generation speed.

The mental model

Think of sculpting, but in reverse. You start with a masterpiece (a clean image). The 'forward process' is like repeatedly hitting it with a tiny hammer, adding a little dust each time, until it's just a cloud of dust (pure noise). The model's job, the 'reverse process,' is to learn how to perfectly undo each of those steps, reconstructing the masterpiece from the dust cloud. To generate a new image, you just start with a random cloud of dust and run this learned reverse process.

How it works

The process has two parts. First, the forward diffusion process is a fixed procedure where you take an image and add a small amount of Gaussian noise over many steps (e.g., 1000). At the end, the image is indistinguishable from random noise. Second, the reverse process trains a neural network to do the opposite. At each step, the model is given a noisy image and a timestep, and it learns to predict the noise that was added. By subtracting this predicted noise, it takes one step back towards a cleaner image. To generate a new image from scratch, you start with pure random noise and apply this trained de-noising model iteratively for all the steps until a clean image emerges. It is a latent variable model because the noisy image at each step is a latent representation of the final clean image.

When to use it

Use diffusion models when the absolute highest quality and diversity of generation is the top priority, and you can tolerate slow, computationally expensive inference. This is ideal for offline asset generation, such as creating artwork, synthetic data for training other models, or generating frames for an animated sequence.

When not to use it

Avoid diffusion models for applications requiring real-time or low-latency generation, like interactive video filters. The multi-step sampling process makes them inherently slower than single-shot models like GANs. Their high computational and memory requirements also make them difficult to deploy on resource-constrained devices like mobile phones.

One canonical example

Text-to-image generation is the most famous application. A model like Stable Diffusion takes a text prompt (e.g., 'an astronaut riding a horse') and uses it to guide the reverse diffusion process. Instead of just de-noising randomly, the model de-noises in a way that steers the emerging image towards the concepts described in the text prompt. This guidance allows for powerful, controllable image synthesis from natural language.