Normalizing Flows: Shaping Simple Distributions into Complex Ones

Why it exists

Many generative models like GANs can create realistic data but cannot tell you the probability of a specific data point. Other models provide only an approximation. Normalizing flows were developed to create generative models that can compute the exact probability density of the data, which is critical for many statistical tasks.

The mental model

A normalizing flow is a sequence of invertible transformations. Imagine taking a simple, known probability distribution, like a uniform blob of points, and applying a series of learned "stretches" and "squishes". These operations "flow" the simple distribution into a shape that matches your complex target data distribution, such as images of faces. Because each step is invertible, you can always reverse the process.

How it works

A normalizing flow uses the change of variables formula from statistics to transform a simple distribution into a complex one. A model, f, maps a simple variable, z, from a known distribution (like a Gaussian) to a data point, x = f(z). The formula allows for computing the exact probability of x by using the probability of z and the determinant of the Jacobian of the inverse transformation. This requires the transformation to be invertible and its Jacobian determinant to be easy to compute, which heavily constrains the neural network design.

When to use it

Use a normalizing flow when you need a generative model that provides exact, tractable likelihoods. This is crucial for tasks like anomaly detection (low likelihood points are anomalies), data compression, and when you need a principled way to evaluate how well your model fits the data. They are used in flow-based generative models to explicitly model a probability distribution.

When not to use it

Avoid normalizing flows if your primary goal is just sample generation and you don't need exact likelihoods; other models are often more efficient for that specific task. The architectural constraints (invertibility, tractable Jacobian) can make them less expressive or slower to train than other generative models for a given parameter count.

One canonical example

A flow-based model like Glow can be trained on a dataset of faces. After training, you can sample a random vector from a simple Gaussian distribution, pass it through the trained flow, and generate a new, realistic-looking face. Crucially, you can also take an existing image of a face, pass it backward through the flow, and get its exact log-probability under the model's learned distribution.