Latent Diffusion Models (LDM)

WHY IT EXISTS Diffusion models generate images by learning to reverse a gradual noising process, but doing that directly on raw pixels is expensive. A single 512 by 512 image has hundreds of thousands of pixel values, and a useful model needs dozens of denoising steps, each one a full pass over that many values. Latent diffusion models exist to make that process affordable by moving it into a much smaller space.

THE MENTAL MODEL Think of a pretrained autoencoder as a lossy but perceptually faithful compressor, like sculpting a detailed miniature instead of a life size statue. It squeezes an image down into a small grid that keeps the structure a viewer actually notices and discards the pixel level detail that does not matter perceptually. All of the expensive iterative denoising happens on that miniature, and only at the very end does a decoder blow it back up to full resolution.

HOW IT WORKS An encoder, trained as part of a variational autoencoder, maps an image into a latent grid downsampled by a large factor in each spatial dimension, commonly eight times smaller per side. Noise is added to that latent over a forward diffusion process, and a U-Net is trained to predict and remove it step by step in reverse, optionally conditioned on a text embedding injected through cross attention layers inside the U-Net. Once denoising finishes, a decoder reconstructs the final latent into a full resolution pixel image in a single pass.

WHEN IT MATTERS This is the difference between a diffusion model that needs a data center and one that runs on a single consumer GPU, which is why text to image tools became widely usable at all. The footgun is that the autoencoder is lossy: fine detail the compressor was never trained to preserve precisely, small text, exact finger counts, distant faces, can degrade during compression, and there is no way for the diffusion process to recover information it never saw.

ONE CONCRETE EXAMPLE Generating a 512 by 512 image directly in pixel space means denoising a tensor with over 750,000 values at every one of fifty steps. A latent diffusion model instead encodes the image into a 64 by 64, four channel latent, denoises that far smaller tensor for fifty steps conditioned on a text prompt, and decodes the result into the final 512 by 512 image exactly once, cutting the compute of every intermediate step by roughly two orders of magnitude.