Diffusion

From picture to noise

Image generators such as Stable Diffusion and FLUX learn to make pictures by first learning how to destroy them. Take a picture and mix in a little random noise. Then a little more. After enough steps nothing of the picture is left, only noise.

The mixing follows a fixed formula. At each time t the picture keeps a share of its signal and gets a share of noise, and the two shares are planned in advance by a schedule. No learning is needed for this direction.

Try it. Move the time slider from the clean picture to pure noise.

The same on a cloud of points

A picture with thousands of pixels is a point in a space with thousands of dimensions. To watch what happens we use a space we can draw, two dimensions. Each training example is now one point, and together they draw a heart.

Adding noise shakes every point. At the end the heart has melted into a round blob, the same blob whatever shape we started with.

Try it. Move the time slider and watch the shape dissolve.

Learning to undo a little noise

The model is trained on one task. Given a noisy example and the time t, guess the clean example, or equivalently the noise that was added. It sees millions of examples at every noise level until its guesses are good.

The arrows show the guess at every place of the plane. With a lot of noise they all point toward the middle of the data. With little noise they point to the nearest part of the heart. For this tiny dataset we can compute the perfect guess exactly, so the arrows are the ideal a real network tries to learn.

Try it. Move the time and watch the arrows change.

From noise to a shape, step by step

To create something new, start from pure random noise and run the process backwards. Ask the model for its guess, take a small step toward it, add back the right amount of noise for the next time, and repeat. Out of the blob the shape slowly appears.

More steps give cleaner results and take longer. Early generators needed about a thousand steps. Better samplers such as DDIM, used here, get good pictures in 20 to 50, and distilled models in four or fewer.

Try it. Generate with 50 steps, then with 3.

Steering with a prompt

Our data now holds two shapes, hearts and stars. A model that is also told the class, or a whole text prompt in a real generator, can make the one you ask for. Here the conditional guess looks only at the asked class.

Generators push even harder with classifier-free guidance. They compute the guess with and without the prompt and step further in the direction of the difference. A higher guidance scale follows the prompt more strictly, and too high a scale overshoots the shape.

Try it. Ask for stars, then raise the guidance scale.

Back to pictures

For pictures the steps are the same, only the points have 256 numbers instead of two. Our toy knows three tiny 16 by 16 pictures, and its perfect denoiser brings back one of them from noise. A perfect denoiser of a small dataset can only copy its training data. A real network makes slightly imperfect guesses that blend what it learned, and that is how it paints pictures nobody drew before.

Stable Diffusion saves work by denoising not the pixels but a compressed latent picture. In its first version a 512 by 512 color image becomes a 64 by 64 grid with 4 channels, 48 times fewer numbers, and a decoder turns the final latent back into pixels. Newer generators such as Stable Diffusion 3 and FLUX.1 keep this idea but replace the U-Net denoiser with a transformer.

Training task

Guess the noise added to an example at time t

Generation

Start from noise, remove it in 20 to 50 steps

Prompt

Text encoder plus classifier-free guidance

Latent diffusion

Stable Diffusion, 2022, works on 64 × 64 × 4 latents

Denoiser today

A diffusion transformer (Stable Diffusion 3, FLUX.1)