Generative AI for Images: Diffusion Models Explained

Ask an image generator for "a lighthouse at dawn in watercolour" and it starts with something that looks nothing like a lighthouse: random noise. Over a few dozen steps, the noise is reshaped into a picture. That reversal of noise is what diffusion models learn.

The idea sounds strange until you see the maths, which is mostly addition. This guide walks through it with a small NumPy demonstration, then explains how it grows into the text-to-image systems people use.

The forward process: destroying an image on purpose

Training starts by ruining images. Take a clean image, add a little Gaussian noise, then a little more, for many steps, until nothing of the original is left. The DDPM paper that made diffusion practical used 1,000 steps, with the amount of noise added per step rising linearly from 0.0001 to 0.02 and pixel values scaled from -1 to 1 (Ho et al.: DDPM).

A useful property makes this cheap. You do not need to apply 500 small steps to get the image at step 500. A closed-form formula jumps straight there: keep a known fraction of the image and add the matching amount of fresh noise (Nichol and Dhariwal: Improved DDPM).

How quickly the image disappears is set by the noise schedule. The script compares two: DDPM's linear schedule and the cosine schedule proposed in the improved DDPM paper.

import numpy as np

# A 16 x 16 greyscale "image": a bright ring on a dark background, scaled to [-1, 1]
yy, xx = np.mgrid[0:16, 0:16]
radius = np.hypot(yy - 7.5, xx - 7.5)
x0 = np.where(np.abs(radius - 5) < 1.5, 1.0, -1.0)

T = 1000
betas = np.linspace(1e-4, 0.02, T)                      # the linear schedule used in the DDPM paper
alpha_bar = np.cumprod(1 - betas)                        # how much of the original survives after t steps
s = 0.008                                                # the cosine schedule from Improved DDPM
f = np.cos((np.arange(T + 1) / T + s) / (1 + s) * np.pi / 2) ** 2
alpha_bar_cos = (f / f[0])[1:]

rng = np.random.default_rng(0)
noise = rng.standard_normal(x0.shape)


def noisy(t, ab=alpha_bar):
    # jump straight to step t: keep sqrt(alpha_bar) of the image, add sqrt(1 - alpha_bar) of noise
    a = ab[t - 1]
    return np.sqrt(a) * x0 + np.sqrt(1 - a) * noise


def show(img):
    shades = " .:-=+*#%@"
    levels = np.clip((img + 2) / 4 * len(shades), 0, len(shades) - 1).astype(int)
    return ["".join(shades[v] for v in row) for row in levels]


pictures = [show(noisy(t)) if t else show(x0) for t in (0, 100, 300, 1000)]
print("step:      0                 100               300               1000")
for rows in zip(*pictures):
    print("   " + "  ".join(rows))

print(f"\n{'step':>5} {'signal kept (linear)':>21} {'signal kept (cosine)':>21} {'match with original':>20}")
for t in (1, 50, 100, 250, 500, 750, 1000):
    match = np.corrcoef(noisy(t).ravel(), x0.ravel())[0, 1]
    print(f"{t:>5} {np.sqrt(alpha_bar[t - 1]):>21.3f} {np.sqrt(alpha_bar_cos[t - 1]):>21.3f} {match:>20.2f}")

# The network's training task: given a noisy image and its step, predict the noise that was added.
# With the exact noise, the original comes back perfectly; with a 10% error, how wrong is it?
print(f"\n{'step':>5} {'error in the recovered image, noise predicted 10% wrong':>58}")
for t in (100, 500, 900, 1000):
    a = alpha_bar[t - 1]
    guess = noise + 0.1 * rng.standard_normal(x0.shape)
    recovered = (noisy(t) - np.sqrt(1 - a) * guess) / np.sqrt(a)
    print(f"{t:>5} {np.abs(recovered - x0).mean():>58.2f}")

step:      0                 100               300               1000
   ::::::::::::::::  ::-:::--:.:: :.:  --=-:=++: :- -.:  +=*+-+%#-.-+ =.-
   ::::::::::::::::  :::-:-::-::.::.:  ::=+-*:=+-..:-.-  -=*#=%-+#+-:-+:=
   :::::######:::::  :-:::##%*%%-::-=  -=-=:*%@=@@=-:*#  =*++-=*%.%%*+=%@
   :::##########:::  =-:*##*###**#.=:  *+==*#=###=++.*:  @%+.+*.***:--:@-
   :::##########:::  ::-%#+##%*%**-:=  =:*@#:*#%+@=++-#  +=%%* +*#-@.-#+@
   ::####::::####::  ::#**#--:-#%#*:-  -:+==#=+.**@++-+  +-=:.**%-@=%--+#
   ::###::::::###::  ::**#-:.-.:#+#::  -:--#+-.+ :#:#:-  +-..*#=:#.-* +-+
   ::###::::::###::  ::#*%-----:*#*.:  --#+@=++=+--*+ -  =+*-%*##*#+.=-.+
   ::###::::::###::  :.*##-:--- %##::  :.=##+-+*+ %##==  -::++%=#%# %+*++
   ::###::::::###::  ::+#*-::.::*##.   =::*++:-.:--#*.   += =-#=+:-=.+=: 
   ::####::::####::  -:*#%#:.--%#%#-:  =:+*@*- *+%*%#=-  *=-=@=+.@##+#+*=
   :::##########:::  .-.##*####*%#:..   + **=#*+#+@#:    .# ==:*=-*-%+- .
   :::##########:::  -::%***#***#%::.  +-:@=++#++-#%:-   #==@.--*--.*#-+.
   :::::######:::::  :-:=:####**+:.:-  :*-#.#*#*+=@- :=  -%+@-*=*=-:@= -*
   ::::::::::::::::  :--::.:.--.... .  :*+-:.: +* . . .  -%#=-:-.#%.: : :
   ::::::::::::::::  -:-:=::=:.::-.--  +:+:*-:%:.--+.=+  %=#-@+=@=.+=#:*#

 step  signal kept (linear)  signal kept (cosine)  match with original
    1                 1.000                 1.000                 1.00
   50                 0.985                 0.996                 0.98
  100                 0.947                 0.986                 0.94
  250                 0.724                 0.920                 0.69
  500                 0.280                 0.703                 0.23
  750                 0.058                 0.380                 0.01
 1000                 0.006                 0.000                -0.04

 step    error in the recovered image, noise predicted 10% wrong
  100                                                       0.03
  500                                                       0.27
  900                                                       4.17
 1000                                                      13.07

What the output shows

• The ring fades into static. At step 100 it is still clear, by step 300 it is faint, and at step 1000 nothing of it remains.
• The linear schedule destroys information early. By step 500 only 28% of the original signal is left, and by step 750 under 6%. The improved DDPM paper made the same observation: the last quarter of the linear schedule is almost pure noise, wasting steps. The cosine schedule keeps 70% at step 500 and spreads the destruction more evenly.
• One jump back does not work. The network's job is to predict the noise in a noisy image. If its prediction is 10% wrong, recovering the image directly from step 100 gives a tiny error of 0.03. From step 1000 the same mistake gives an error of 13.07, far larger than the image's own range of -1 to 1. Undoing the noise in one jump multiplies the network's mistakes.

That last point is why generation takes many steps. The sampler removes a little noise, asks the network again about the slightly cleaner image, and repeats, so errors are corrected before they grow.

What the network learns

During training, the network sees a noisy image, the step number, and has to predict the noise that was added. The true answer is known, because training made the noise. The loss is simply how far the prediction is from it.

At generation time, the process runs backwards. Start from pure noise, predict the noise, remove part of it, and repeat down to step 0. A network that has learned to denoise every level of noise has, in effect, learned what images look like.

From pixels to prompts

The demonstration works on pixels. Real image generators add three ideas.

Latent diffusion. Running diffusion on full-resolution pixels is expensive. Latent diffusion first compresses images with a pretrained autoencoder and runs the diffusion process in that smaller latent space, then decodes the result. The paper behind Stable Diffusion showed this keeps quality while cutting the compute dramatically (Rombach et al.: latent diffusion).

Text conditioning. The same paper adds cross-attention layers so the denoising network can attend to an encoding of the prompt. Each denoising step is then pushed towards images that match the text. See attention explained.

Guidance. Classifier-free guidance trains one model both with and without the prompt, and at generation time exaggerates the difference between the two predictions. Raising the guidance scale makes images follow the prompt more closely at the cost of variety (Ho and Salimans: classifier-free guidance).

Newer variants

The field has moved on from the original recipe without abandoning its core. Many recent models replace the convolutional U-Net with a transformer that works on patches of the latent image. Some, such as the models described in the Stable Diffusion 3 paper, use rectified flow, which connects data and noise along a straight line. That paper reports better results than established diffusion formulations for high-resolution text-to-image generation (Esser et al.: rectified flow transformers).

The libraries reflect this variety. Hugging Face's Diffusers library, for example, ships interchangeable schedulers. Its DDPM scheduler defaults to the original 1,000 steps and linear schedule, with a cosine option among the alternatives (Hugging Face: DDPMScheduler).

What this explains in practice

For how image, audio and text models are being combined, see multimodal AI. For how language models generate text, a different process, see how LLMs work.

Common mistakes

• Thinking the model stores or collages training images. It learns to predict noise and generates new images from that.
• Assuming more steps always help. Past a point, extra steps change little and cost time.
• Turning guidance up to the maximum. Very high guidance tends to give oversaturated, distorted images.
• Comparing results across different seeds when judging a settings change.
• Ignoring the licence of an open image model before using its output commercially.

Modern generative models, including image generation, are covered in the Generative AI course.

Sources

• Denoising Diffusion Probabilistic Models (arXiv) — Ho et al.
• Improved Denoising Diffusion Probabilistic Models (arXiv) — Nichol and Dhariwal
• High-Resolution Image Synthesis with Latent Diffusion Models (arXiv) — Rombach et al.
• Classifier-Free Diffusion Guidance (arXiv) — Ho and Salimans
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv) — Esser et al.
• DDPMScheduler (Hugging Face Diffusers documentation) — Hugging Face