Skip to main content

Latent Diffusion & Stable Diffusion

While DDPMs are powerful, they are extremely slow and computationally expensive because they operate directly on pixels. For a 512×512512 \times 512 image, the network has to process 262,144 pixels at every step.

Stable Diffusion solved this by moving the process into Latent Space.

1. The Perceptual Compression (The VAE)​

Instead of diffusing the image itself, we use a Variational Autoencoder (VAE) to compress the image into a smaller "latent" representation zz.

  • Pixel Space: 512×512×3512 \times 512 \times 3 (Huge)
  • Latent Space: 64×64×464 \times 64 \times 4 (Small)

The Diffusion process happens entirely in this compressed space. Once we've denoised the latent vector, the VAE Decoder turns it back into a high-resolution image. This makes Stable Diffusion 8x - 64x faster than pixel-based models.

2. Conditioning: The CLIP Bridge​

How does the model know to draw a "Cyberpunk Forest"? It uses Cross-Attention.

  1. Text Encoding: Your prompt is fed into a text encoder (CLIP).
  2. Tokens to Vectors: The text is converted into a sequence of mathematical vectors.
  3. Attention Mechanism: At every layer of the U-Net, the image-latents "look" at the text-vectors to decide which pixels should become trees and which should become neon lights.

3. Classifier-Free Guidance (CFG)​

CFG is a trick to make the model follow your prompt more strictly. During generation, the model predicts the noise twice:

  1. Once with the prompt.
  2. Once without the prompt (using empty text).

The model then pushes the generation away from the unconditioned version and towards the prompted version.

The CFG Scale: A higher scale makes the image more vibrant and "correct" to the prompt, but too high can make the image look burnt or distorted.

4. The Ecosystem​

Stable Diffusion isn't just one model; it's a framework that supports:

TechniquePurpose
ControlNetAdds spatial control (e.g., following a human pose or a sketch).
LoRAFine-tunes the model on a specific style or character with very little data.
InpaintingRepairs or replaces specific parts of an image.
Textual InversionTeaches the model a new concept by finding its "word vector."
Summary

Stable Diffusion = (Latent Space Diffusion) + (CLIP Text Conditioning) + (Iterative Refinement). It is the standard for open-source generative art today.