Probability & Statistics: The Language of Noise

Diffusion models don't just "generate" images; they learn the underlying probability distribution of data. To understand how we can turn white noise into a masterpiece, we first need to master the mathematics of uncertainty.

---

1. Variance ($\sigma^2$) and Covariance ($\Sigma$)

In the context of Diffusion, Variance is our measure of "destruction."

• Low Variance: The data points are tightly clustered. In an image, this means pixels are predictable and structured.
• High Variance: The data points are scattered. As we add noise to an image, we are effectively increasing its variance until the original structure is lost in a cloud of entropy.

Covariance tells us how pixel $A$ changes in relation to pixel $B$. Natural images have high covariance (neighboring pixels are usually similar). Pure noise has zero covariance—each pixel is a "lone wolf."

---

2. The Gaussian Distribution (The "Bell Curve")

Diffusion models rely almost exclusively on the Multivariate Gaussian Distribution. Why? Because of the Central Limit Theorem: when you add up many independent random effects (like noise), the result converges to a Gaussian.

The probability density function (PDF) is defined as:

$\mathcal{N}(x; \mu, \sigma^2 I) = \frac{1}{\sqrt{(2\pi)^d |\sigma^2 I|}} \exp\left( -\frac{1}{2} (x-\mu)^T (\sigma^2 I)^{-1} (x-\mu) \right)$

In simple terms:

• $\mu$ (Mean): The "center" of our data (e.g., the original image).
• $\sigma^2$ (Variance): The "spread" of the noise we've added.

---

3. The Reparameterization Trick

This is the "Secret Sauce" that makes training possible.

If we want to sample $x$ from a distribution $\mathcal{N}(\mu, \sigma^2)$, we usually can't backpropagate through a random sampling operation.

The Solution: Express the random variable $x$ as a deterministic function of a "standard" noise variable $\epsilon \sim \mathcal{N}(0, I)$:

$x = \mu + \sigma \cdot \epsilon$

Now, the randomness is isolated in $\epsilon$, and we can compute gradients with respect to $\mu$ and $\sigma$ perfectly!

---

4. Sum of Gaussians (The "Nice" Property)

Diffusion models involve adding noise over thousands of steps. Do we need to simulate every single step to get to step $T$? No.

Because the sum of two Gaussians is also a Gaussian, we can jump from the original image $x_0$ to any noisy state $x_t$ in a single calculation:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) I)$

The Golden Rule: This property allows us to train on any arbitrary timestep $t$ without having to run the simulation linearly. It's what makes Diffusion training efficient.

---

Summary Checklist

Concept	Role in Diffusion
Variance	Controls the "noise level" at each step.
Gaussian PDF	Defines the transition between clean and noisy states.
Reparameterization	Allows the neural network to learn via Backpropagation.
Linearity	Enables "jumping" to any timestep $t$ during training.