MicDZ's Blog

Echoes and Gradients: The Hidden Unity Between Radar Imaging and Deep Learning

2025-12-20T07:29:29.000Z

At first glance, a passive radar system and a Deep Neural Network (DNN) seem to inhabit different worlds. One deals with wave physics, antennas, and scattering coefficients; the other deals with tensors, activation functions, and gradient descent.

However, if you strip away the domain-specific jargon, you find that the algorithm used to form an image from radar echoes (Back-Projection) and the algorithm used to train a neural network (Back-Propagation) are, at their core, the same mathematical operation.

They are both applications of the Adjoint Operator to solve an Inverse (Reconstruction) Problem.

In this post, we will explore this duality, using the specific example of a passive imaging system to illuminate the inner workings of deep learning.

The Forward Pass: Scattering vs. Inference

To understand the backward process, we must first look at the forward process—how data generates an output.

The Imaging System

Imagine a passive radar system with multiple receivers (Rx) to detect active targets.

Transmission: The transmission process follows the physics of electromagnetic wave propagation, which can be modeled by the Green’s function $\mathcal{H}(\mathbf r)$ . In this article, we use the 2D free-space Green’s function: $\mathcal{H}(\mathbf r) = \dfrac{1}{\sqrt{j\pi |\mathbf r|}}\exp(j2\pi|\mathbf r|/\lambda)$ where $\lambda$ is the wavelength, and $|\mathbf r|$ is the distance from the transmitter to the observation point.
Target: The target emits an electromagnetic wave, which propagates through space, denoted as the $s(\mathbf r)$ .
Recieption: The receivers collect the waves from the target. The received signal $g(\mathbf{r})$ is a convolution of the transmitted wave and the Green’s function: $g(\mathbf r) = s(\mathbf r) * \mathcal{H}(\mathbf r)$ where $*$ denotes convolution over space $\mathbf r$ .

Following is an interactive demo of Green’s function and the conjugate Green’s function.

The Neural Network

Now, look at a single connection in a neural network between an input neuron and an output neuron.

Input: An activation value passes from the previous layer.
Interaction: The value is multiplied by a Weight $\theta$ .
Output: The result contributes to the next layer.

Mathematically, the output $\hat{y}$ is the input $x$ modulated by the model weights $\theta$ :

\hat{y} = \pi_\theta(x)

The loss function $\mathcal{L}(\hat{y}, y)$ measures the difference between the predicted output $\hat{y}$ and the true label $y$ . Our goal is to minimize this loss by adjusting the weights $\theta$ .

\min_{\theta} \sum \mathcal{L}(\pi_\theta(x), y)

The Inverse Problem: “Who is Responsible?”

In both fields, we face an inverse problem:

Imaging: Time Reversal

How do we find the target? We take the signal received at the Rx and perform Time Reversal. We mathematically “play the tape backward.”
We project the received signal back into space, applying a negative time delay corresponding to the distance from Rx to the target, and then from the target to Tx.

Back-Proprogation in Imaging

Now, the imaging problem: we have the received data $g(\mathbf r)$ , and we want to find the target $s(\mathbf r)$ .How do we do it? We mathematically “propagate” the received waves back into space to see where they intersect. This is often called Matched Filtering or Back-projection. Mathematically, we convolve the received signal with the complex conjugate of the Green’s function, $h^*(\mathbf r)$ :

\begin{aligned}\hat s(\mathbf r) &= g(\mathbf r) * \mathcal{H}^*(\mathbf r) \\&= \left[ s(\mathbf r) * \mathcal{H}(\mathbf r) \right] * \mathcal{H}^*(\mathbf r) \\&= s(\mathbf r) * \left[ \mathcal{H}(\mathbf r) * \mathcal{H}^*(\mathbf r) \right] \\&= s(\mathbf r) * \delta(\mathbf r) \\&= s(\mathbf r)\end{aligned}

The physical meaning of $\mathcal{H}^*(\mathbf r)$ is that we are reversing the wave propagation process. By convolving with the conjugate Green’s function, we assume the sensor behaves like a transmitter in reverse, sending a signal back through the same medium.

It is notable that this is an ideal model; in practice we can not perfectly capture the whole received signal in space, and noise will be present. But the core idea remains: by back-projecting the received signals using the conjugate of the forward operator, we can reconstruct the target location.

Deep Learning: Gradient Descent

How do we find the optimal weight? We can use Gradient Descent. However, calculating the gradient of the loss function with respect to each weight directly is computationally expensive. Instead, we use the Back-Propagation algorithm.

Consider a simple neural network with $N$ layers:

Input tensor: $\alpha^{(0)}=x$ .
Parameters at layer $l$ : Parameters matrix $W^{(l)}$ and bias vector $b^{(l)}$ .
Pre-activation at layer $l$ : $z^{(l)} = W^{(l)} \alpha^{(l-1)} + b^{(l)}$ .
Nonlinear activation at layer $l$ : $\alpha^{(l)} = \sigma(z^{(l)})$ .
Output tensor: $\hat{y} = \alpha^{(N)} = \pi_\theta(x)$ .

Then, the loss function is $L_\theta(\hat{y}, y) = \mathcal{L}(\alpha^{(N)}, y)$ . And we want to compute the gradient of the loss $\nabla_\theta \mathcal{L}$ which contains $\frac{\partial \mathcal{L}}{\partial W^{(l)}}$ and $\frac{\partial \mathcal{L}}{\partial b^{(l)}}$ for all layers $l$ .

BP introduces intermediate variables called Error Signals $\delta^{(l)}$ , defined as:

\delta^{(l)} \equiv\dfrac{\partial \mathcal{L}}{\partial z^{(l)}}

Error signal at the output layer $N$ :
Applying the chain rule, we can express the error signal at layer $l$ in terms of the error signal at layer $l+1$ :
$\frac{\partial \mathcal{L}}{\partial z_i^{(N)}} = \sum_j \frac{\partial \mathcal{L}}{\partial a_j^{(N)}} \frac{\partial a_j^{(N)}}{\partial z_i^{(N)}}$
Because the activation function $\sigma$ is applied element-wise, we have:
$\frac{\partial a_j^{(N)}}{\partial z_i^{(N)}} = \sigma'(z_i^{(N)}) \delta_{ij}$
where $\delta_{ij} = 1$ if $i=j$ and $0$ otherwise. Therefore, we can simplify the expression for the error signal:
$\delta_i^{(N)} = \frac{\partial \mathcal{L}}{\partial z_i^{(N)}} = \frac{\partial \mathcal{L}}{\partial a_i^{(N)}} \sigma'(z_i^{(N)})$
To simplify notation, we can express this in vector form:
$\delta^{(N)} = \frac{\partial \mathcal{L}}{\partial a^{(N)}} \odot \sigma'(z^{(N)})$
where $\odot$ denotes the element-wise (Hadamard) product.
Error signal at hidden layers $l < N$ :
Again, applying the chain rule, we have:
$\frac{\partial \mathcal{L}}{\partial z_i^{(l)}} = \sum_{k} \frac{\partial \mathcal{L}}{\partial z_k^{(l+1)}} \frac{\partial z_k^{(l+1)}}{\partial z_i^{(l)}}$
According to the definition of $\delta_k^{(l+1)} = \frac{\partial \mathcal{L}}{\partial z_k^{(l+1)}}$ , we can rewrite the above equation as:
$\frac{\partial \mathcal{L}}{\partial z_i^{(l)}} = \sum_{k} \delta_k^{(l+1)} \frac{\partial z_k^{(l+1)}}{\partial z_i^{(l)}}$
$\frac{\partial z_k^{(l+1)}}{\partial z_i^{(l)}}$ can be computed using the Jacobian matrix:
$\left[ \frac{\partial z^{(l+1)}}{\partial z^{(l)}} \right]_{ki} = \frac{\partial z_k^{(l+1)}}{\partial z_i^{(l)}}$
Thus, we can express the error signal at layer $l$ as:
$\delta^{(l)} = \left( \frac{\partial z^{(l+1)}}{\partial z^{(l)}} \right)^T \delta^{(l+1)}$
Since $z^{(l+1)} = W^{(l+1)} \alpha^{(l)} + b^{(l+1)}$ and $\alpha^{(l)} = \sigma(z^{(l)})$ , we have:
$\delta^{(l)} = \left( W^{(l+1)} \right)^T \delta^{(l+1)} \odot \sigma'(z^{(l)})$

Together with (15) and (10), we can recursively compute the error signals from the output layer back to the input layer. This is the essence of the Back-Propagation algorithm.

After computing the error signals for all layers, we can compute the gradients with respect to the weights and biases:

\frac{\partial \mathcal{L}}{\partial b_i^{(l)}} = \frac{\partial \mathcal{L}}{\partial z_i^{(l)}} \frac{\partial z_i^{(l)}}{\partial b_i^{(l)}}

because $z_i^{(l)}= W_i^{(l)} \alpha^{(l-1)} + b_i^{(l)}$ , we have $\frac{\partial z_i^{(l)}}{\partial b_i^{(l)}} = 1$ . Therefore:

\frac{\partial \mathcal{L}}{\partial b_i^{(l)}} = \delta_i^{(l)}

Vectorizing this, we get:

\frac{\partial \mathcal{L}}{\partial b^{(l)}} = \delta^{(l)}

Similarly, for the weights:

\frac{\partial \mathcal{L}}{\partial W_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial z_i^{(l)}} \frac{\partial z_i^{(l)}}{\partial W_{ij}^{(l)}}

Since $z_i^{(l)} = W_i^{(l)} \alpha^{(l-1)} + b_i^{(l)}$ , we have $\frac{\partial z_i^{(l)}}{\partial W_{ij}^{(l)}} = \alpha_j^{(l-1)}$ . Therefore:

\frac{\partial \mathcal{L}}{\partial W_{ij}^{(l)}} = \delta_i^{(l)} \alpha_j^{(l-1)}

In matrix form, this becomes:

\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \delta^{(l)} \left( \alpha^{(l-1)} \right)^T

Following is an interactive demo of Back-Propagation in Deep Learning. The task is to fit a simple 2D function:

f(x_1, x_2) = \begin{cases} 1 & \text{if } x_1 + x_2 > 1 \\ 0 & \text{otherwise} \end{cases}

The Synthesis: Adjoint Symmetry

The striking similarity between the equations of wave imaging and neural network training is not a coincidence. Both systems are governed by the same mathematical principle: the duality between a linear operator and its adjoint.

To see this clearly, let us unify our notation. In both cases, we have a Forward Operator that maps an internal state (the target or the weights ) to an observation (the signal or the prediction ).

The Unified Forward Map

In a linearized or local sense, both processes can be viewed as an operator acting on a hidden source to produce a measurable result:

Feature	Passive Radar (Imaging)	Deep Learning (Training)
Hidden Source	Target Reflectivity $s(\mathbf{r})$	Parameter Update $\Delta \theta$
Forward Operator	Green’s Function Convolution $\mathcal{H}$	Jacobian Matrix $J = \frac{\partial \pi}{\partial \theta}$
Observation	Received Echoes $g(\mathbf{r})$	Prediction Error $\Delta y = (\hat{y} - y)$

Mathematically, we represent the forward pass as:

y_{obs} = A x

where $A$ is our forward transformation.

The Inverse Problem and the Adjoint

The “Inverse Problem” asks: given the observation $y_{obs}$ , what was the source $x$ ?

Solving this perfectly usually requires computing the inverse $A^{-1}$ , which is often computationally impossible or “ill-posed” (highly sensitive to noise). Instead, both fields use the Adjoint Operator $A^\dagger$ (the complex conjugate transpose in matrix terms).

In Radar: Back-Projection

The reconstructed image $\hat{s}$ is formed by applying the adjoint of the propagation operator:

\hat{s} = \mathcal{H}^\dagger g

As derived earlier, $\mathcal{H}^\dagger$ corresponds to convolution with the conjugate Green’s function $h^*(\mathbf{r})$ . Physically, this is Time Reversal.

In Deep Learning: Back-Propagation

The gradient used to update weights is found by applying the adjoint of the network’s local sensitivity:

\nabla_\theta \mathcal{L} = \left( \frac{\partial \hat{y}}{\partial \theta} \right)^T \delta

Here, the transpose $(W^{(l)})^T$ is the adjoint of the forward weight matrix $W^{(l)}$ . Just as the radar signal is sent back through the channel, the error signal is “sent back” through the weights.

Why the Adjoint Works

The effectiveness of both algorithms stems from the Adjoint Identity:

\langle Ax, y \rangle = \langle x, A^\dagger y \rangle

In imaging, this means the energy we collect at the receivers ( $\langle \text{signal}, \text{echo} \rangle$ ) is mathematically equivalent to the energy at the target location when we project our “knowledge” back into the scene.

In deep learning, this identity ensures that the most efficient way to reduce the loss in the high-dimensional output space is to “project” that error back onto the weights.

Conclusion: One Algorithm, Two Realities

The “Back” in both Back-Projection and Back-Propagation refers to the same fundamental movement: reversing the flow of information.

In Radar, we reverse Time to locate a target in physical space.
In Deep Learning, we reverse Causality to locate the source of an error in parameter space.

Whether we are “seeing” a stealth aircraft through passive echoes or “learning” to recognize a cat in a photo, we are performing the same mathematical ritual: calculating the forward transformation, measuring the discrepancy, and using the Adjoint Operator to project that discrepancy back to its origin.

Deep learning, in this light, is not just a branch of computer science; it is a form of Computational Imaging where the “scene” being reconstructed is the internal logic of the model itself.

From DDPM to DDIM — Accelerate Sampling without Extra Training

2025-10-29T00:36:41.000Z

Introduction

Although Denoising Diffusion Probabilistic Models (DDPM) have shown great success in generative tasks, their sampling process is often slow due to the need for many iterative steps (e.g., 1000 steps), for each requiring a forward pass through the neural network.

To address this issue, researchers from Stanford University proposed Denoising Diffusion Implicit Models (DDIM), which allow for accelerated sampling without the need for additional training from a pre-trained DDPM model. DDIM achieve this by introducing a non-Markovian diffusion process that enables fewer sampling steps while maintaining high-quality outputs.

Key Concepts

DDIM follow a similar notation to DDPM, where $x_t$ represents the noisy data at time step $t$ , and $\epsilon_\theta(x_t, t)$ is the neural network predicting the noise component. The key difference lies in the sampling process.

p_\theta(\mathrm x_0) = \int p_\theta(\mathrm x_{0:T}) \mathrm d\mathrm x_{1:T}

where

p_\theta(\mathrm x_{0:T}) = p(\mathrm x_T) \prod_{t=1}^{T} p_\theta(\mathrm x_{t-1}|\mathrm x_t)

DDPM use a special property of the forward process:

q(\mathrm x_{t}|\mathrm x_0) = \mathcal N(\mathrm x_t; \sqrt{\bar \alpha_t} \mathrm x_0, (1-\bar \alpha_t) \mathbf I)

so that $\mathrm x_t$ can be reparameterized as:

\mathrm x_t = \sqrt{\bar \alpha_t} \mathrm x_0 + \sqrt{1-\bar \alpha_t} \epsilon, \quad \epsilon \sim \mathcal N(0, \mathbf I)

Therefore, the posterior distribution (which the model learns to predict) can be derived as:

\begin{aligned}q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) &= \frac{q(\mathrm x_t|\mathrm x_{t-1}, \mathrm x_0) q(\mathrm x_{t-1}|\mathrm x_0)}{q(\mathrm x_t|\mathrm x_0)} \\&= \mathcal N\left(\mathrm x_{t-1}; \tilde \mu_t(\mathrm x_t, \mathrm x_0), \tilde \beta_t \mathbf I\right)\end{aligned}

the derivation of $\tilde \mu_t(\mathrm x_t, \mathrm x_0)$ and $\tilde \beta_t$ relies on the Markovian assumption and the reparameterization of $\mathrm x_t$ .

The key idea of DDIM is to generalize this posterior distribution to a non-Markovian form with arbitrary transition steps.

Forward Process

The forward process of DDPM is defined as:

q(\mathrm x_{1:T}|\mathrm x_0) = \prod_{t=1}^{T} q(\mathrm x_t|\mathrm x_{t-1})

where

q(\mathrm x_t|\mathrm x_{t-1}) = \mathcal N(\mathrm x_t; \sqrt{1-\beta_t} \mathrm x_{t-1}, \ \beta_t \mathbf I)

What we are going to use is the property that we can directly sample $\mathrm x_t$ from $\mathrm x_0$ :

q(\mathrm x_t|\mathrm x_0) = \mathcal N(\mathrm x_t; \sqrt{\bar \alpha_t} \mathrm x_0, (1-\bar \alpha_t) \mathbf I)

Consider an arbitrary set of time steps $k,s \in \{0, 1, \ldots, T\}$ with $s \leq k - 1$ . We want to find the distribution $q(\mathrm x_s|\mathrm x_k, \mathrm x_0)$ , so that we can skip some steps during sampling.

However, unlike the Markovian case, we cannot derive this distribution directly. Instead, we make an assumption that it follows a Gaussian distribution:

q(\mathrm x_s |\mathrm x_k, \mathrm x_0) = \mathcal N(k\mathrm x_0 + m\mathrm x_k, \sigma^2 \mathbf I)

where $k, m, \sigma$ are coefficients to be determined. We need to solve these three coefficients.

\begin{aligned}x_s &= k \mathrm x_0 + m \mathrm x_k + \sigma \epsilon, \quad \epsilon \sim \mathcal N(0, \mathbf I) \\&= k \mathrm x_0 + m(\sqrt{\bar \alpha_k} \mathrm x_0 + \sqrt{1-\bar \alpha_k} \epsilon') + \sigma \epsilon, \quad \epsilon' \sim \mathcal N(0, \mathbf I) \\&= (k + m \sqrt{\bar \alpha_k}) \mathrm x_0 + m \sqrt{1-\bar \alpha_k} \epsilon' + \sigma \epsilon\\&\sim \mathcal N\left((k + m \sqrt{\bar \alpha_k}) \mathrm x_0, m^2 (1-\bar \alpha_k) + \sigma^2\right)\end{aligned}

To ensure consistency with the forward process, we need to match the mean and variance with those of $q(\mathrm x_s|\mathrm x_0)$ :

q(\mathrm x_s|\mathrm x_0) = \mathcal N(\sqrt{\bar \alpha_s} \mathrm x_0, (1-\bar \alpha_s) \mathbf I)

This gives us two equations:

\begin{aligned}k + m \sqrt{\bar \alpha_k} &= \sqrt{\bar \alpha_s} \\m^2 (1-\bar \alpha_k) + \sigma^2 &= 1-\bar \alpha_s\end{aligned}

We treat $\sigma$ as a hyperparameter to control the stochasticity of the sampling process. By solving the above equations, we can obtain:

\begin{aligned}m &= \sqrt{\frac{(1-\bar \alpha_s) - \sigma^2}{1-\bar \alpha_k}} \\k &= \sqrt{\bar \alpha_s} - m \sqrt{\bar \alpha_k}\end{aligned}

With these coefficients, we can define the DDIM sampling step from $\mathrm x_k$ to $\mathrm x_s$ as:

\begin{aligned}q(\mathrm x_s |\mathrm x_k, \mathrm x_0) &= \mathcal N\left(\mathrm x_s; k \mathrm x_0 + m \mathrm x_k, \sigma^2 \mathbf I\right)\\&= \mathcal N\left(\mathrm x_s; \sqrt{\bar \alpha_s} \mathrm x_0 + \sqrt{(1-\bar \alpha_s) - \sigma^2} \cdot \frac{\mathrm x_k - \sqrt{\bar \alpha_k} \mathrm x_0}{\sqrt{1-\bar \alpha_k}}, \sigma^2 \mathbf I\right)\end{aligned}

The magnitude of $\sigma$ controls how stochastic the forward process is; when $\sigma \to 0$ , we reach an extreme case where as long as we observe $x_0$ and $x_t$ for some $t$ , then $x_{t-1}$ becomes known and fixed.

The forward process of DDIM indeed changes the forward process from a Markovian one to a non-Markovian one, and makes an assumption on the form of the transition distribution, this causing a doubt on whether the learned model can still work well under this new forward process.

However, what DDPM truly use in the forward process is the reparameterization trick $\mathrm x_t = \sqrt{\bar \alpha_t} \mathrm x_0 + \sqrt{1-\bar \alpha_t} \epsilon$ , which is still valid in DDIM since the forward process of DDIM doesn’t change this property, and thus the learned model can still be used in DDIM sampling, which we will explain in the next section.

Sampling Process

As we have trained a DDPM model to predict $\epsilon_\theta(\mathrm x_t, t)$ , we can use it to estimate $\mathrm x_{t-1}$ from $\mathrm x_t$ :

\begin{aligned}\mathrm x_{t-1} &= \frac{1}{\sqrt{\alpha_t}}\left(\mathrm x_t - \frac{1-\alpha_t}{\sqrt{1-\bar \alpha_t}} \epsilon_\theta(\mathrm x_t, t)\right) + \sigma_t \epsilon, \quad \epsilon \sim \mathcal N(0, \mathbf I)\\\end{aligned}

Essentially, what we have learned in DDPM is to predict $\mathrm x_{t-1}$ from $\mathrm x_t$ :

p_\theta(\mathrm x_{t-1}|\mathrm x_t) = \mathcal N\left(\mathrm x_{t-1}; \tilde \mu_\theta(\mathrm x_t, t), \sigma_t^2 \mathbf I\right)

The goal of DDPM is to minimize the difference between $p_\theta(\mathrm x_{t-1}|\mathrm x_t)$ and $q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)$ . However, DDPM perform the reparameterization of $\mathrm x_t$ in terms of $\mathrm x_0$ and $\epsilon$ :

\mathrm x_t = \sqrt{\bar \alpha_t} \mathrm x_0 + \sqrt{1-\bar \alpha_t} \epsilon, \quad \epsilon \sim \mathcal N(0, \mathbf I)

Thus, in essence, we learn the noise $\epsilon$ to predict $\mathrm x_0$ from $\mathrm x_t$ in DDPM:

\mathrm x_0 = \frac{\mathrm x_t - \sqrt{1-\bar \alpha_t} \epsilon}{\sqrt{\bar \alpha_t}}

With the learned $\epsilon$ , during the sampling process of DDIM, we can substitute the predicted noise $\epsilon_\theta(\mathrm x_t, t)$ for $\epsilon$ to give an estimate of $\mathrm x_0$ :

\mathrm{\hat{x}}_0(\mathrm{x}_k, k) = \frac{\mathrm{x}_k - \sqrt{1-\alpha_k} \mathrm{\epsilon}_\theta(\mathrm{x}_k, k)}{\sqrt{\alpha_k}}

Then we can generalize this to predict $\mathrm x_s$ from $\mathrm x_k$ :

\begin{aligned}\mathrm x_s &= \sqrt{\bar \alpha_s} \mathrm {\hat{x}}_0 + \sqrt{(1-\bar \alpha_s) - \sigma^2} \cdot \frac{\mathrm x_k - \sqrt{\bar \alpha_k} \mathrm {\hat{x}}_0}{\sqrt{1-\bar \alpha_k}} + \sigma \epsilon, \quad \epsilon \sim \mathcal N(0, \mathbf I)\\&= \sqrt{\bar \alpha_s} \left(\frac{\mathrm x_k - \sqrt{1-\bar \alpha_k} \epsilon_\theta(\mathrm x_k, k)}{\sqrt{\bar \alpha_k}}\right) + \sqrt{(1-\bar \alpha_s) - \sigma^2} \cdot \frac{\mathrm x_k - \sqrt{\bar \alpha_k} \left(\frac{\mathrm x_k - \sqrt{1-\bar \alpha_k} \epsilon_\theta(\mathrm x_k, k)}{\sqrt{\bar \alpha_k}}\right)}{\sqrt{1-\bar \alpha_k}} + \sigma \epsilon\\&= \sqrt{\bar \alpha_s} \left(\frac{\mathrm x_k - \sqrt{1-\bar \alpha_k} \epsilon_\theta(\mathrm x_k, k)}{\sqrt{\bar \alpha_k}}\right) + \sqrt{(1-\bar \alpha_s) - \sigma^2} \cdot \sqrt{1-\bar \alpha_k} \cdot \frac{\epsilon_\theta(\mathrm x_k, k)}{\sqrt{1-\bar \alpha_k}} + \sigma \epsilon\\&= \sqrt{\bar \alpha_s} \left(\frac{\mathrm x_k - \sqrt{1-\bar \alpha_k} \epsilon_\theta(\mathrm x_k, k)}{\sqrt{\bar \alpha_k}}\right) + \sqrt{(1-\bar \alpha_s) - \sigma^2} \cdot \epsilon_\theta(\mathrm x_k, k) + \sigma \epsilon\end{aligned}

By iteratively applying this process with a reduced number of steps, we can efficiently generate high-quality samples from the diffusion model without the need for additional training.

The Choice of σ

When $\sigma = 0$ , the sampling process becomes deterministic, meaning that for a given initial noise input, the output will always be the same. This can be beneficial in scenarios where reproducibility is important, or when we want to ensure that the generated samples are consistent.

On the other hand, when $\sigma > 0$ , the sampling process introduces stochasticity, allowing for more diverse outputs from the same initial noise input. This can be advantageous in creative applications where variability is desired, such as in image generation or other generative tasks.

When $\sigma = \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t}$ , the sampling process of DDIM is equivalent to that of DDPM.

To prove this, we can substitute this value of $\sigma$ into the DDIM sampling equation:

\begin{aligned}\mathrm x_{t-1} &= \sqrt{\bar \alpha_{t-1}} \left(\frac{\mathrm x_t - \sqrt{1-\bar \alpha_t} \epsilon_\theta(\mathrm x_t, t)}{\sqrt{\bar \alpha_t}}\right) + \sqrt{(1-\bar \alpha_{t-1}) - \sigma^2} \cdot \epsilon_\theta(\mathrm x_t, t) + \sigma \epsilon \\&= \frac{\sqrt{\bar \alpha_{t-1}}}{\sqrt{\bar \alpha_t}} \mathrm x_t - \frac{\sqrt{\bar \alpha_{t-1}} \sqrt{1-\bar \alpha_t}}{\sqrt{\bar \alpha_t}} \epsilon_\theta(\mathrm x_t, t) + \sqrt{(1-\bar \alpha_{t-1}) - \frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t} \cdot \epsilon_\theta(\mathrm x_t, t) + \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t} \epsilon \\&= \frac{1}{\sqrt{\alpha_t}}\left(\mathrm x_t - \frac{1-\alpha_t}{\sqrt{1-\bar \alpha_t}} \epsilon_\theta(\mathrm x_t, t)\right) + \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t} \epsilon\\&= \tilde \mu_\theta(\mathrm x_t, t) + \sigma_t \epsilon\end{aligned}

where $\sigma_t = \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t}$ which is exactly the same as the $\tilde \beta_t = \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t}$ in DDPM.

DDIM introduce a hyperparameter $\eta$ to control the level of stochasticity in the sampling process. The relationship between $\sigma$ and $\eta$ is defined as:

\sigma_t = \eta \sqrt{\frac{1-\bar \alpha_{t-1}}{1-\bar \alpha_{t}}\beta_t}

When $\eta = 0$ , the sampling process is deterministic, while when $\eta = 1$ , it becomes equivalent to the stochastic sampling of DDPM. By adjusting $\eta$ , users can control the trade-off between sample diversity and fidelity according to their specific needs.

Guide to Diffusion Denoising Probabilistic Models (DDPM)

2025-10-24T18:35:14.000Z

Introduction

Diffusion Denoising Probabilistic Models (DDPM) have emerged as a powerful class of generative models in machine learning. They have shown remarkable capabilities in generating high-quality images, audio, and other data types. In this guide, we will derive the fundamental concepts behind DDPMs, explain their mathematical foundations, and provide insights into their implementation.

Prior Knowledge

Probabilistic Notations

Before diving into DDPMs, let’s clarify some probabilistic notations that will be used throughout this guide:

$p(x)$ : Probability density function (PDF) of a random variable $x$ .

If we want to calculate the probability that $x$ falls within a specific range, we integrate $p(x)$ over that range.

P(a \leq x \leq b) = \int_{a}^{b} p(x) \, \mathrm{d}x

where $P$ is the probability measure.

$p(x|y)$ : Conditional probability density function of $x$ given $y$ .

p(x|y) = \frac{p(x, y)}{p(y)}

$\mathbb{E}[f(x)]$ : Expectation of random variable $f(x)$ .

\mathbb{E}[f(x)] = \int p(x) f(x) \, \mathrm dx

$\mathbb{E}_{p}[f(x)]$ : Expectation of function $f(x)$ with respect to the distribution $p(x)$ .

\mathbb{E}_{q}[f(x)] = \int q(x) f(x) \, \mathrm dx

$p(x)=\mathcal{N}(x;\mu, \sigma^2)$ indicates that $X\sim p(x)$ follows a Gaussian distribution with mean $\mu$ and variance $\sigma^2$ . The variable before the semicolon is the random variable, and the variables after the semicolon are the parameters of the distribution.

Bayes’ Theorem

Bayes’ theorem is a fundamental concept in probability theory that relates conditional probabilities. It states that:

p(x|y) = \frac{p(y|x)p(x)}{p(y)}

If we center our attention on $x$ , we can interpret Bayes’ theorem as follows:

$p(x|y)$ is the posterior probability of $x$ given $y$ , which represents our updated belief about $x$ after observing $y$ .
$p(y|x)$ is the likelihood of observing $y$ given $x$ , which indicates how probable the observation $y$ is for a specific value of $x$ .
$p(x)$ is the prior probability of $x$ , which represents our initial belief about $x$ before observing $y$ .
$p(y)$ is the marginal likelihood of $y$ , which serves as a normalizing constant to ensure that the posterior probability sums to 1.

Example:
Suppose there is a rare disease that affects 1% of the population. A diagnostic test for this disease has a 99% accuracy rate, which means if someone has the disease, the test will correctly identify them as positive 99% of the time. While, if someone does not have the disease, the test will incorrectly identify them as positive 2% of the time, calling it a false positive. If a person tests positive, what is the probability that they actually have the disease?
Let $D$ be the event that the person has the disease, and $T$ be the event that the test result is positive. We want to find $p(D|T)$ .
Let’s define the known probabilities:
$p(D) = 0.01$ (prior probability of having the disease)
$p(\neg D) = 0.99$ (prior probability of not having the disease)
$p(T|D) = 0.99$ (likelihood of testing positive given the disease)
$p(T|\neg D) = 0.02$ (likelihood of testing positive given no disease)
Using Bayes’ theorem, we can calculate the posterior probability $p(D|T)$ :
$p(D|T) = \frac{p(T|D)p(D)}{p(T)}$
To find $p(T)$ , we can use the law of total probability:
$p(T) = p(T|D)p(D) + p(T|\neg D)p(\neg D)$
this gives us the total probability of testing positive, considering both cases (having the disease and testing positive, not having the disease and testing positive).
Now we can plug in the values:
$p(T) = (0.99)(0.01) + (0.02)(0.99) = 0.0099 + 0.0198 = 0.0297$
Now we can calculate $p(D|T)$ :
$p(D|T) = \frac{(0.99)(0.01)}{0.0297} = \frac{0.0099}{0.0297} \approx 0.3333$
Therefore, if a person tests positive, there is approximately a 33.33% chance that they actually have the disease, despite the high accuracy of the test. This counterintuitive result highlights the importance of considering prior probabilities when interpreting test results.

Additive Property of Gaussian Distribution

The Gaussian (or normal) distribution is a continuous probability distribution characterized by its mean $\mu$ and variance $\sigma^2$ . A key property of Gaussian distributions is that the sum of two independent Gaussian random variables is also a Gaussian random variable. Specifically, if we have two independent Gaussian random variables:

\begin{aligned} X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2)\\X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2)\end{aligned}

Then, their sum $Y = X_1 + X_2$ is also Gaussian distributed:

Y \sim \mathcal{N}(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)

if $Y = X_1 - X_2$ , then:

Y \sim \mathcal{N}(\mu_1 - \mu_2, \sigma_1^2 + \sigma_2^2)

We can extend this property to any linear combination of independent Gaussian random variables. For example, if we have:

Y = aX_1 + bX_2 + c

where $a$ , $b$ , and $c$ are constants, then $Y$ is also Gaussian distributed:

Y \sim \mathcal{N}(a\mu_1 + b\mu_2 + c, a^2\sigma_1^2 + b^2\sigma_2^2)

Kullback-Leibler (KL) Divergence

Kullback-Leibler (KL) divergence is a measure of how one probability distribution diverges from a second, expected probability distribution. It is often used in creating a loss function for training probabilistic models, if we want to train a model to approximate a target distribution $p(x)$ with a model distribution $q(x)$ , we can minimize the KL divergence from $p$ to $q$ :

D_{KL}(p || q) = \int p(x) \log \frac{p(x)}{q(x)} \, \mathrm{dx} = \mathbb{E}_{p} \left[ \log \frac{p(x)}{q(x)} \right]

The KL divergence of Gaussian distributions has a closed-form solution. If we have two Gaussian distributions:

\begin{aligned} p(x) = \mathcal{N}(\mu_p, \Sigma_p^2)\\q(x) = \mathcal{N}(\mu_q, \Sigma_q^2)\end{aligned}

\begin{aligned} p(x) = \frac{1}{(2\pi)^{k/2} |\Sigma_p|^{1/2}} \exp\left(-\frac{1}{2}(x - \mu_p)^T \Sigma_p^{-1} (x - \mu_p)\right)\\q(x) = \frac{1}{(2\pi)^{k/2} |\Sigma_q|^{1/2}} \exp\left(-\frac{1}{2}(x - \mu_q)^T \Sigma_q^{-1} (x - \mu_q)\right)\end{aligned}

where $|\cdot|$ denotes the determinant of a matrix, we plug these PDFs into the KL divergence formula and simplify to get:

D_{KL}(p || q) = \mathbb{E}_{p} \left[ \log \frac{p(x)}{q(x)} \right]

with three parts:

Part 1:

\frac{1}{2} \left(\log |\Sigma_q| - \log |\Sigma_p|\right)

Part 2:

\frac{1}{2} \mathbb{E}_{p} \left[ (x - \mu_q)^T \Sigma_q^{-1} (x - \mu_q) \right]

Since $(x-\mu_p)^T \Sigma_p^{-1} (x-\mu_p)$ is actually a scalar, for a scalar, its trace is itself. Therefore, we can rewrite Part 2 as:

\frac{1}{2} \mathbb{E}_{p} \left[ \text{tr}\left((x - \mu_p)^T \Sigma_p^{-1} (x - \mu_p)\right) \right]

Using the cyclic property of the trace operator $\text{tr}(ABC) = \text{tr}(CAB) = \text{tr}(BCA)$ , we can further simplify it to:

\begin{aligned}&\frac{1}{2} \mathbb{E}_{p} \left[ \text{tr}\left(\Sigma_p^{-1} (x - \mu_p)(x - \mu_p)^T\right) \right]\\=& \frac{1}{2} \text{tr}\left(\Sigma_p^{-1} \mathbb{E}_{p} \left[ (x - \mu_p)(x - \mu_p)^T \right] \right)\\=& \frac{1}{2} \text{tr}\left(\Sigma_p^{-1} \Sigma_p \right) = \frac{1}{2} \text{tr}(I) = \frac{k}{2}\end{aligned}

Part 3:

\begin{aligned}&\frac{1}{2} \mathbb{E}_{p} \left[ (x-\mu_p + \mu_p - \mu_q)^T \Sigma_q^{-1} (x-\mu_p + \mu_p - \mu_q) \right] \\=& \frac{1}{2} \mathbb{E}_{p} \left[\text{tr}\left((x-\mu_p + \mu_p - \mu_q)^T \Sigma_q^{-1} (x-\mu_p + \mu_p - \mu_q)\right) \right] \\=& \frac{1}{2} \mathbb{E}_{p} \left[\text{tr}\left(\Sigma_q^{-1} (x-\mu_p + \mu_p - \mu_q)(x-\mu_p + \mu_p - \mu_q)^T\right) \right] \\=& \frac{1}{2} \text{tr}\left(\Sigma_q^{-1} \mathbb{E}_{p} \left[ (x-\mu_p + \mu_p - \mu_q)(x-\mu_p + \mu_p - \mu_q)^T \right] \right) \\=& \mathbb{E}_{p}\!\left[(x-\mu_p)(x-\mu_p)^T\right]+ (\mu_p-\mu_q)(\mu_p-\mu_q)^T+ \underbrace{\mathbb{E}_{p}\!\left[(x-\mu_p)(\mu_p-\mu_q)^T\right]}_{=\,0}+ \underbrace{\mathbb{E}_{p}\!\left[(\mu_p-\mu_q)(x-\mu_p)^T\right]}_{=\,0}\\=& \frac{1}{2} \text{tr}\left(\Sigma_q^{-1} \mathbb{E}_{p} \left[ (x-\mu_p)(x-\mu_p)^T + (\mu_p - \mu_q)(\mu_p - \mu_q)^T \right] \right) \\=& \frac{1}{2} \text{tr}\left(\Sigma_q^{-1} \Sigma_p\right) + \frac{1}{2} \text{tr}((\mu_p - \mu_q)^T \Sigma_q^{-1} (\mu_p - \mu_q))\\=& \frac{1}{2} \text{tr}\left(\Sigma_q^{-1} \Sigma_p\right) + \frac{1}{2} (\mu_p - \mu_q)\Sigma_q^{-1}(\mu_p - \mu_q)^T \\\end{aligned}

Combining all three parts, we obtain the closed-form expression for the KL divergence between two Gaussian distributions:

D_{KL}(p || q) = \frac{1}{2} \left(\log |\Sigma_q| - \log |\Sigma_p|\right) + \frac{k}{2} + \frac{1}{2} \text{tr}\left(\Sigma_q^{-1} \Sigma_p\right) + \frac{1}{2} (\mu_p - \mu_q)\Sigma_q^{-1}(\mu_p - \mu_q)^T

The Big Picture

The goal of DDPM is to learn a generative model that can produce samples from a target data distribution $p_{\text{data}}(z)$ . Here we need to define what is “data” and what is “generation”.

“Data” refers to the real-world samples we want our model to learn from, such as images, audio, or text. These samples are drawn from an unknown distribution $p_{\text{data}}(z)$ that we aim to approximate. What we have access to are finite samples $z_1, z_2, \ldots, z_N$ from this distribution, which we use to train our model.

“Generation” refers to the process of creating plausible new samples $z_{\text{new}}$ from the learned model. I use the term “plausible” because the generated samples should resemble the real data in terms of structure and characteristics, but they do not need to be exact replicas of any specific training sample. The goal is to capture the underlying distribution of the data so that the model can produce diverse and realistic samples. In the term of mathematics, we want to learn a model distribution $p_{\theta}(z)$ that approximates the true data distribution $p_{\text{data}}(z)$ , and we can generate new samples $z_{\text{new}}$ by sampling from $p_{\theta}(z)$ , which makes $p_{\text{data}}(z_{\text{new}})$ not too small.

Overview of Diffusion Denoising Probabilistic Models (DDPM)

The core idea of DDPM is to model the data generation process as a gradual denoising of a noisy input. The model consists of two main processes: the forward diffusion process and the reverse denoising process.

What the model actually learns is the reverse denoising process, which starts from pure noise and iteratively denoises it to produce a clean sample that resembles the training data. This process is parameterized by a neural network that predicts the noise component at each step, allowing the model to gradually refine the noisy input into a high-quality output.

Reverse Denoising Process

Let’s denote the clean data sample as $\mathrm x_0$ and the noisy sample at step $t$ as $\mathrm x_t$ . At the final step $T$ , $\mathrm x_T$ is essentially pure noise $p(\mathrm x_T) = \mathcal{N}(\mathrm x_T;\mathbf 0, \mathbf I)$ .

p_\theta(\mathrm x_{0:T}) := p(\mathrm x_T) \prod_{t=1}^{T} p_\theta(\mathrm x_{t-1}|\mathrm x_t)

where $\mathrm x_{0:T}$ denotes the sequence of all latent variables from the initial clean sample to the final noisy sample. This formulation captures the entire generative process, starting from pure noise $\mathrm x_T$ and progressively denoising it to obtain the clean sample $\mathrm x_0$ , using a markovian structure where each step depends only on the previous one.

If we fixed a $\mathrm x_0$ , there may be infinite possible $\mathrm x_{1:T}$ that can lead to this $\mathrm x_0$ . Therefore, to get the probability of generating $\mathrm x_0$ , we need to marginalize out all possible $\mathrm x_{1:T}$ :

p_\theta(\mathrm x_0) = \int p_\theta(\mathrm x_{0:T}) \, \mathrm d \mathrm x_{1:T}

For each denoising step, we model the conditional probability $p_\theta(\mathrm x_{t-1}|\mathrm x_t)$ as a Gaussian distribution:

p_\theta(\mathrm x_{t-1}|\mathrm x_t) = \mathcal{N}(\mathrm x_{t-1}; \mu_\theta(\mathrm x_t, t), \Sigma_\theta(\mathrm x_t, t))

where $\mu_\theta(\mathrm x_t, t)$ and $\Sigma_\theta(\mathrm x_t, t)$ are the mean and covariance predicted by a neural network parameterized by $\theta$ at step $t$ .

Forward Diffusion Process

The forward diffusion process gradually adds noise to the clean data sample $\mathrm x_0$ over $T$ steps, resulting in a sequence of noisy samples $\mathrm x_1, \mathrm x_2, \ldots, \mathrm x_T$ . This process is defined as:

q(\mathrm x_{1:T}|\mathrm x_0) := \prod_{t=1}^{T} q(\mathrm x_t|\mathrm x_{t-1})

where each step adds Gaussian noise to the previous sample:

q(\mathrm x_t|\mathrm x_{t-1}) = \mathcal{N}(\mathrm x_t; \sqrt{1 - \beta_t} \mathrm x_{t-1}, \beta_t \mathbf I)

Here, $\beta_t$ is a variance schedule that controls the amount of noise added at each step. As $t$ increases, more noise is added, and by the final step $T$ , the sample $\mathrm x_T$ becomes nearly indistinguishable from pure Gaussian noise.

However, for training purposes, we don’t want to sample $\mathrm x_t$ sequentially from $\mathrm x_0$ to $\mathrm x_T$ , this cost too much time. Fortunately, due to the properties of Gaussian distributions, we can directly sample $\mathrm x_t$ from $\mathrm x_0$ without going through all intermediate steps.

Proof:
Since each step in the forward diffusion process is Gaussian, the composition of these Gaussian steps results in another Gaussian distribution. As Fomula (29) shows, we can derive the mean and variance of $q(\mathrm x_t|\mathrm x_0)$ by recursively applying the properties of Gaussian distributions:
$\mathrm x_t = \sqrt{1 - \beta_t} \mathrm x_{t-1} + \sqrt{\beta_t} \epsilon_{t-1}, \quad \epsilon_{t-1} \sim \mathcal{N}(\mathbf 0, \mathbf I)$
where $\sqrt{1 - \beta_t} \mathrm x_{t-1}$ contribute to the mean, and $\sqrt{\beta_t} \epsilon_{t-1}$ contribute to the variance.
$\mathrm x_t = \sqrt{\bar \alpha_t} \mathrm x_0 + \sqrt{1 - \bar \alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(\mathbf 0, \mathbf I)$
By recursively substituting $\mathrm x_{t-1}$ back to $\mathrm x_0$ , we can express $\mathrm x_t$ directly in terms of $\mathrm x_0$ and the accumulated noise:
$\begin{aligned}\mathrm x_t &= \sqrt{1 - \beta_t} \left( \sqrt{1 - \beta_{t-1}} \mathrm x_{t-2} + \sqrt{\beta_{t-1}} \epsilon_{t-2} \right) + \sqrt{\beta_t} \epsilon_{t-1} \\&= \sqrt{(1 - \beta_t)(1 - \beta_{t-1})} \mathrm x_{t-2} + \sqrt{(1 - \beta_t)\beta_{t-1}} \epsilon_{t-2} + \sqrt{\beta_t} \epsilon_{t-1} \\&\vdots \\&= \sqrt{\prod_{s=1}^{t} (1 - \beta_s)} \mathrm x_0 + \sum_{s=1}^{t} \sqrt{\beta_s \prod_{u=s+1}^{t} (1 - \beta_u)} \epsilon_{s-1}\\&\sim \mathcal{N}\left(\sqrt{\bar \alpha_t} \mathrm x_0, \left(1 - \bar \alpha_t\right) \mathbf I\right)\end{aligned}$
where $\alpha_t = 1 - \beta_t$ and $\bar \alpha_t = \prod_{s=1}^{t} \alpha_s$ .

So, we have:

q(\mathrm x_t|\mathrm x_0) = \mathcal{N}(\mathrm x_t; \sqrt{\bar \alpha_t} \mathrm x_0, (1 - \bar \alpha_t) \mathbf I)

Loss Function

To train the DDPM, we need to define a loss function that measures how well the model’s predicted denoising matches the true denoising process. We can use the variational lower bound (VLB) on the negative log-likelihood of the data as our loss function:

\begin{aligned}\mathbb E[-\log p_\theta(\mathrm x_0)] &= \mathbb E[-\log \int p_\theta(\mathrm x_{0:T}) \, \mathrm d \mathrm x_{1:T}] \\&= \mathbb E\left[-\log \int q(\mathrm x_{1:T}|\mathrm x_0) \frac{p_\theta(\mathrm x_{0:T})}{q(\mathrm x_{1:T}|\mathrm x_0)} \, \mathrm d \mathrm x_{1:T}\right] \\&= \mathbb E\left[-\log \mathbb E_{q} \left[\frac{p_\theta(\mathrm x_{0:T})}{q(\mathrm x_{1:T}|\mathrm x_0)}\right]\right] \\&= -\log \mathbb E_q \left[ \frac{q(\mathrm x_{1:T}|\mathrm x_0)}{p_\theta(\mathrm x_{0:T})}\right] \\\end{aligned}

Because $\log$ is a concave function, we can apply Jensen’s inequality to obtain the variational lower bound:

\begin{aligned}\mathbb E_q[-\log p_\theta(\mathrm x_0)] &\leq \mathbb E_q\left[-\log \frac{p_\theta(\mathrm x_{0:T})}{q(\mathrm x_{1:T}|\mathrm x_0)}\right] := \mathcal{L}(\theta)\\\end{aligned}

However, this expression is still intractable due to the integral over all possible $\mathrm x_{1:T}$ . To make it tractable, we can decompose the loss into a sum of KL divergences at each time step:

\begin{aligned}\mathcal{L}(\theta)&= \mathbb E_q\left[-\log p(\mathrm x_T) - \sum_{t\geq 1} \log p_\theta(\mathrm x_{t-1}|\mathrm x_t) + \sum_{t\geq 1} \log q(\mathrm x_t|\mathrm x_{t-1})\right] \\&= \mathbb E_q\left[-\log p(\mathrm x_T) - \sum_{t>1} \log \frac{p_\theta(\mathrm x_{t-1}|\mathrm x_t)}{q(\mathrm x_t|\mathrm x_{t-1})} - \log \frac{p_\theta(\mathrm x_0|\mathrm x_1)}{q(\mathrm x_1|\mathrm x_0)}\right] \\\end{aligned}

Then, we use the Bayes’ theorem to rewrite the $q(\mathrm x_t|\mathrm x_{t-1})$ term:

q(\mathrm x_t|\mathrm x_{t-1}) = \frac{q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) q(\mathrm x_t|\mathrm x_0)}{q(\mathrm x_{t-1}|\mathrm x_0)}

This gives us:

\begin{aligned}&\mathcal{L}(\theta) = \mathbb E_q\left[-\log p(\mathrm x_T) - \sum_{t>1} \log \frac{p_\theta(\mathrm x_{t-1}|\mathrm x_t) q(\mathrm x_{t-1}|\mathrm x_0)}{q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) q(\mathrm x_t|\mathrm x_0)} - \log \frac{p_\theta(\mathrm x_0|\mathrm x_1)}{q(\mathrm x_1|\mathrm x_0)}\right] \\\end{aligned}

Rearranging the terms, we can cancel out $q(\mathrm x_{t-1}|\mathrm x_0)$ and $q(\mathrm x_t|\mathrm x_0)$ in the summation:

\begin{aligned}\sum_{t>1} \log \frac{q(\mathrm x_{t-1}|\mathrm x_0)}{ q(\mathrm x_t|\mathrm x_0)}&=\log q(\mathrm x_1|\mathrm x_0) - \log q(\mathrm x_2|\mathrm x_0) + \log q(\mathrm x_2|\mathrm x_0) - \log q(\mathrm x_3|\mathrm x_0) + \ldots+ \log q(\mathrm x_{T-1}|\mathrm x_0) - \log q(\mathrm x_T|\mathrm x_0)\\&=\log q(\mathrm x_1|\mathrm x_0) - \log q(\mathrm x_T|\mathrm x_0)\end{aligned}

So the loss function becomes:

\begin{aligned}\mathcal{L}(\theta)&= \mathbb E_q\left[-\log p(\mathrm x_T) + \sum_{t>1} \log \frac{q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)}{p_\theta(\mathrm x_{t-1}|\mathrm x_t)} + \log q(\mathrm x_T|\mathrm x_0) - \log p_\theta(\mathrm x_0|\mathrm x_1) + \log q(\mathrm x_1|\mathrm x_0)\right] \\&= \mathbb E_q\left[-\log \frac{p(\mathrm x_T)}{q(\mathrm x_T|\mathrm x_0)} - \sum_{t>1} \log \frac{p_\theta(\mathrm x_{t-1}|\mathrm x_t)}{q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)} - \log \frac{p_\theta(\mathrm x_0|\mathrm x_1)}{q(\mathrm x_1|\mathrm x_0)}\right] \\&= \mathbb E_q\left[D_{KL}(q(\mathrm x_T|\mathrm x_0) || p(\mathrm x_T)) + \sum_{t>1} D_{KL}(q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) || p_\theta(\mathrm x_{t-1}|\mathrm x_t)) + D_{KL}(q(\mathrm x_1|\mathrm x_0) || p_\theta(\mathrm x_0|\mathrm x_1))\right] \\\end{aligned}

Now, we have decomposed the loss function into three main components:

\mathcal{L}(\theta)=\mathbb E_q\!\Big[\underbrace{D_{KL}(q(\mathrm x_T|\mathrm x_0) \,||\, p(\mathrm x_T))}_{\mathcal L_T}+ \underbrace{\sum_{t>1} D_{KL}(q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) \,||\, p_\theta(\mathrm x_{t-1}|\mathrm x_t))}_{\mathcal L_{t-1}}+ \underbrace{D_{KL}(q(\mathrm x_1|\mathrm x_0) \,||\, p_\theta(\mathrm x_0|\mathrm x_1))}_{\mathcal L_0}\Big]

Since each term in the loss function is a KL divergence between two Gaussian distributions, we can use the closed-form solution of KL divergence for Gaussian distributions to compute the loss efficiently during training. This allows us to optimize the model parameters $\theta$ using gradient descent, ultimately enabling the DDPM to learn to generate high-quality samples from the target data distribution.

For $\mathcal L_T$ :
Since DDPM fixed the $\beta_t$ schedule, so $\mathcal{L}_T$ is not relevant to the optimization, we can ignore it during training.
For $\mathcal L_{t-1}$ :
To compute $\mathcal{L}_{t-1}$ , we need to determine the posterior distribution $q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)$ . Using Bayes’ theorem, we have:

q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) = \frac{q(\mathrm x_t|\mathrm x_{t-1}, \mathrm x_0) q(\mathrm x_{t-1}|\mathrm x_0)}{q(\mathrm x_t|\mathrm x_0)}

Since the forward process is Markovian, we know that $q(\mathrm x_t|\mathrm x_{t-1}, \mathrm x_0) = q(\mathrm x_t|\mathrm x_{t-1})$ . Therefore, we can rewrite the equation as:

q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) = \frac{q(\mathrm x_t|\mathrm x_{t-1}) q(\mathrm x_{t-1}|\mathrm x_0)}{q(\mathrm x_t|\mathrm x_0)}

Combining with Formula (37), (33), we directly plug in the distributions parameters into the PDF of Gaussian distribution:

\begin{aligned}q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) &= \frac{1}{(2\pi)^{d/2} |\beta_t \mathbf I|^{1/2}} \exp\left(-\frac{1}{2}(\mathrm x_t - \sqrt{1 - \beta_t} \mathrm x_{t-1})^T (\beta_t \mathbf I)^{-1} (\mathrm x_t - \sqrt{1 - \beta_t} \mathrm x_{t-1})\right) \\&\quad \cdot \frac{1}{(2\pi)^{d/2} |(1 - \bar \alpha_{t-1}) \mathbf I|^{1/2}} \exp\left(-\frac{1}{2}(\mathrm x_{t-1} - \sqrt{\bar \alpha_{t-1}} \mathrm x_0)^T ((1 - \bar \alpha_{t-1}) \mathbf I)^{-1} (\mathrm x_{t-1} - \sqrt{\bar \alpha_{t- 1}} \mathrm x_0)\right) \\&\quad \cdot \frac{(2\pi)^{d/2} |(1 - \bar \alpha_t) \mathbf I|^{1/2}}{\exp\left(-\frac{1}{2}(\mathrm x_t - \sqrt{\bar \alpha_t} \mathrm x_0)^T ((1 - \bar \alpha_t) \mathbf I)^{-1} (\mathrm x_t - \sqrt{\bar \alpha_t} \mathrm x_0)\right)} \\&\ldots\end{aligned}

After simplifying the expression, we find that the posterior distribution $q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)$ is also a Gaussian distribution:

q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) = \mathcal{N}(\mathrm x_{t-1}; \tilde \mu_t(\mathrm x_t, \mathrm x_0), \tilde \beta_t \mathbf I)

where

\tilde \mu_t(\mathrm x_t, \mathrm x_0) = \frac{\sqrt{\bar \alpha_{t-1}} \beta_t}{1 - \bar \alpha_t} \mathrm x_0 + \frac{\sqrt{\alpha_t} (1 - \bar \alpha_{t-1})}{1 - \bar \alpha_t} \mathrm x_t

and

\tilde \beta_t = \frac{1 - \bar \alpha_{t-1}}{1 - \bar \alpha_t} \beta_t

Now, we can compute the KL divergence $\mathcal{L}_{t-1}$ between the posterior distribution $q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0)$ and the model’s predicted distribution $p_\theta(\mathrm x_{t-1}|\mathrm x_t)$ using the closed-form solution for KL divergence between two Gaussian distributions:

\begin{aligned}\mathcal{L}_{t-1} &= \mathbb{E}_q\Big [D_{KL}(q(\mathrm x_{t-1}|\mathrm x_t,\mathrm x_0) || p_\theta(\mathrm x_{t-1}|\mathrm x_t)) \Big]\\&= \mathbb{E}_q\Big[\frac{1}{2} \left(\log |\Sigma_\theta(\mathrm x_t, t)| - \log |\tilde \beta_t \mathbf I|\right) + \frac{d}{2} + \frac{1}{2} \text{tr}\left(\Sigma_\theta(\mathrm x_t, t)^{-1} \tilde \beta_t \mathbf I\right) + \frac{1}{2} (\tilde \mu_t(\mathrm x_t, \mathrm x_0)\\ &\quad- \mu_\theta(\mathrm x_t, t))^T \Sigma_\theta(\mathrm x_t, t)^{-1} (\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t)) \Big]\\\end{aligned}

DDPM made a simplification by fixing the covariance $\Sigma_\theta(\mathrm x_t, t)$ to $\sigma_t^2 \mathbf I$ , which allows us to ignore the first three terms during optimization because they do not depend on the model parameters $\theta$ . Therefore, the loss term $\mathcal{L}_{t-1}$ simplifies to:

\begin{aligned}\mathcal{L}_{t-1} &= \mathbb{E}_q\Big[\frac{1}{2} (\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t))^T \Sigma_\theta(\mathrm x_t, t)^{-1} (\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t)) + \text{C}\Big]\\&=\mathbb{E}_q\Big[\frac{1}{2} (\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t))^T \frac{1}{\sigma_t^2} \mathbf I (\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t)) + \text{C}\Big]\\&= \mathbb{E}_q\Big[\frac{1}{2 \sigma_t^2} \|\tilde \mu_t(\mathrm x_t, \mathrm x_0) - \mu_\theta(\mathrm x_t, t)\|^2 + \text{C}\Big]\end{aligned}

Applied Fomula (33), we can express $\mathrm x_t$ in terms of $\mathrm x_0$ and the noise component $\epsilon\sim \mathcal{N}(0, \mathbf I)$ :

\mathrm x_t(\mathrm x_0, \epsilon) = \sqrt{\bar \alpha_t} \mathrm x_0 + \sqrt{1 - \bar \alpha_t} \epsilon

Then we plug this into the loss function, using Formula (46):

\begin{aligned}\mathcal{L}_{t-1} - \text{C} &= \mathbb{E}_{\mathrm x_0, \epsilon}\Big[\frac{1}{2 \sigma_t^2} \big\|\tilde \mu_t(\mathrm x_t(\mathrm x_0, \epsilon), \frac{1}{\sqrt{\bar \alpha_t}}(\mathrm x_t(\mathrm x_0, \epsilon) - \sqrt{1 - \bar \alpha_t} \epsilon)) - \mu_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\big\|^2\Big]\\&= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{1}{2 \sigma_t^2} \Big\|\frac{\sqrt{\bar \alpha_{t-1}} \beta_t}{1 - \bar \alpha_t} \frac{1}{\sqrt{\bar \alpha_t}}(\mathrm x_t(\mathrm x_0, \epsilon) - \sqrt{1 - \bar \alpha_t} \epsilon) + \frac{\sqrt{\alpha_t} (1 - \bar \alpha_{t-1})}{1 - \bar \alpha_t} \mathrm x_t(\mathrm x_0, \epsilon) - \mu_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\Big\|^2\Bigg]\\&= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{1}{2 \sigma_t^2} \Big\|\frac{1}{\sqrt{\bar \alpha_t}} \big(\mathrm x_t(\mathrm x_0,\epsilon) - \frac{\beta_t}{\sqrt{1-\bar \alpha_t}}\epsilon\big) - \mu_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\Big\|^2\Bigg]\\\end{aligned}

Here we changed the expectation from $q$ to $\mathrm x_0$ and $\epsilon$ , because currently $\mathrm x_t$ is fully determined by $\mathrm x_0$ and $\epsilon$ .

Previously, the $q$ of $\mathbb E_q$ means for all $\mathrm x_{1:T}$ follow $q(\mathrm x_{1:T}|\mathrm x_0)$ , but now we only need to sample $\mathrm x_0$ from the data distribution and $\epsilon$ from the standard normal distribution to compute $\mathrm x_t$ . This significantly simplifies the training process, as we no longer need to sample the entire sequence of noisy samples $\mathrm x_{1:T}$ .

With the Formula (51), our training goal becomes:

\mu_\theta(\mathrm x_t, t) \approx \frac{1}{\sqrt{\bar \alpha_t}} \left(\mathrm x_t - \frac{\beta_t}{\sqrt{1 - \bar \alpha_t}} \epsilon\right)

then we can reparameterize the neural network to predict the noise component $\epsilon_\theta(\mathrm x_t, t)$ instead of the mean $\mu_\theta(\mathrm x_t, t)$ :

\mu_\theta(\mathrm x_t, t) = \frac{1}{\sqrt{\bar \alpha_t}} \left(\mathrm x_t - \frac{\beta_t}{\sqrt{1 - \bar \alpha_t}} \epsilon_\theta(\mathrm x_t, t)\right)

Now, $\epsilon_\theta$ becomes a function approximator that learns to predict the noise $\epsilon$ added at each step of the diffusion process from the noisy sample $\mathrm x_t$ .

Because we defined $p_\theta(\mathrm x_{t-1}|\mathrm x_t)$ as a Gaussian distribution $\mathcal{N}(\mathrm x_{t-1}; \mu_\theta(\mathrm x_t, t), \Sigma_\theta(\mathrm x_t, t))$ in Formula (27), where a made an assumption that the covariance $\Sigma_\theta(\mathrm x_t, t) = \sigma_t^2 \mathbf I$ is fixed. Therefore, we can now express the mean $\mu_\theta$ in terms of the predicted noise $\epsilon_\theta$ :

\begin{aligned}\mathrm x_{t-1}&=\mu_\theta(\mathrm x_t, t) + \sigma_t \mathbf z, \quad \mathbf z \sim \mathcal{N}(\mathbf 0, \mathbf I) \\&= \frac{1}{\sqrt{\bar \alpha_t}} \left(\mathrm x_t - \frac{\beta_t}{\sqrt{1 - \bar \alpha_t}} \epsilon_\theta(\mathrm x_t, t)\right) + \sigma_t \mathbf z\end{aligned}

With the parameterization of $\epsilon_\theta$ , we can now rewrite the Formula (51) loss function in terms of the predicted noise:

\begin{aligned}\mathcal{L}_{t-1} - \text{C} &= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{1}{2 \sigma_t^2} \Big\|\frac{1}{\sqrt{\bar \alpha_t}} \big(\mathrm x_t(\mathrm x_0, \epsilon) - \frac{\beta_t}{\sqrt{1-\bar \alpha_t}}\epsilon\big) - \mu_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\Big\|^2\Bigg]\\&= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{1}{2 \sigma_t^2} \Big\|\frac{1}{\sqrt{\bar \alpha_t}} \big(\mathrm x_t(\mathrm x_0, \epsilon) - \frac{\beta_t}{\sqrt{1-\bar \alpha_t}}\epsilon\big) - \frac{1}{\sqrt{\bar \alpha_t}} \left(\mathrm x_t(\mathrm x_0, \epsilon) - \frac{\beta_t}{\sqrt{1 - \bar \alpha_t}} \epsilon_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\right)\Big\|^2\Bigg]\\&= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{\beta_t^2}{2 \sigma_t^2\alpha_t(1-\alpha_t)} \Big\|\epsilon - \epsilon_\theta(\mathrm x_t(\mathrm x_0, \epsilon), t)\Big\|^2\Bigg]\\&= \mathbb{E}_{\mathrm x_0, \epsilon}\Bigg[\frac{\beta_t^2}{2 \sigma_t^2\alpha_t(1-\alpha_t)} \Big\|\epsilon - \epsilon_\theta(\sqrt{\bar \alpha_t}\mathrm x_0+\sqrt{1-\bar\alpha_t}\epsilon,t)\Big\|^2\Bigg]\\\end{aligned}

Now, we have all we need to implement the training algorithm for DDPM.

\begin{algorithm}\caption{Training}\begin{algorithmic}\REPEAT    \STATE $\mathbf{x}_0 \sim q(\mathbf{x}_0)$    \STATE $t \sim \text{Uniform}(\{1, \dots, T\})$    \STATE $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$    \STATE Take gradient descent step on         $\nabla_{\boldsymbol{\theta}} \|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon}, t)\|^2$\UNTIL{converged}\end{algorithmic}\end{algorithm}

\begin{algorithm}\caption{Sampling}\begin{algorithmic}\STATE $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$\FOR{$t = T, \dots, 1$}    \IF{$t > 1$}        \STATE $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$    \ELSE        \STATE $\mathbf{z} = \mathbf{0}$    \ENDIF    \STATE $\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\mathbf{x}_t, t)\right) + \sigma_t \mathbf{z}$\ENDFOR\RETURN $\mathbf{x}_0$\end{algorithmic}\end{algorithm}

Waves, Phasors, and Signals — The Math of Interference and Diffraction

2025-10-21T07:48:42.000Z

Introduction

Have you ever wondered why noise-canceling headphones work? Or how a tiny hologram can contain a 3D image? Or even just why you see shimmering rainbow patterns on a soap bubble?

The answer to all of these is wave interference and diffraction. These are fundamental concepts in physics, but today I want to look at them through a different lens: the lens of signal processing. It turns out that the propagation of a wave is one of the most elegant and powerful signal processing systems in the universe.

The Wave as a Complex Signal

In signal processing and physics, we describe a simple wave using complex exponentials, thanks to Euler’s formula: $e^{i\theta} = \cos(\theta) + i\sin(\theta)$ .

We can represent a wave $\psi$ at a position $\mathbf{r}$ and time $t$ as:

\psi(\mathbf{r}, t) = A e^{i(\omega t - \mathbf{k} \cdot \mathbf{r} + \phi_0)}

Let’s break this down:

$A$ is the amplitude (the wave’s “strength”).
$e^{i(\dots)}$ is the phasor. This is the magic part. It’s a vector on the complex plane (a “clock hand”) that rotates.
$\omega$ is the angular frequency (how fast it oscillates in time).
$\mathbf{k}$ is the wave vector (its direction of travel and spatial frequency, where $|\mathbf{k}| = 2\pi/\lambda$ ).
$\phi_0$ is the initial phase.

For the rest of this post, we’ll mostly ignore the time part ( $e^{i\omega t}$ ) since it’s common to all waves from the same source. We’ll just focus on the complex amplitude (the spatial part):

\psi(\mathbf{r}) = A e^{-i(\mathbf{k} \cdot \mathbf{r} - \phi_0)}

Think of this $\psi(\mathbf{r})$ as a single complex number for every point in space. Its magnitude $A$ is the amplitude, and its angle $e^{-i\phi}$ is its phase.

Propagation is just a phase shift. As a wave travels from a source at $\mathbf{r}_0$ to a point $\mathbf{P}$ , it covers a distance $d = |\mathbf{P} - \mathbf{r}_0|$ . This journey “rotates” the phasor. The wave at $\mathbf{P}$ is:

\psi(\mathbf{P}) = \frac{A_0}{d} e^{-ikd}

(The $A_0/d$ shows the amplitude decaying with distance $d$ , and $k = 2\pi/\lambda$ ).

The key takeaway: Propagation = Phase Rotation.

Interference: The Superposition Problem

Interference is the simplest case. It’s just what happens when two or more waves meet at the same point. The math is simple: you just add them. This is the Principle of Superposition.

Let’s model the classic double-slit experiment. We have two tiny sources (slits) at $\mathbf{r}_1$ and $\mathbf{r}_2$ . We want to know the total wave $\psi_{total}$ at an observation point $\mathbf{P}$ on a screen.

Wave 1 (from slit 1): Propagates a distance $d_1 = |\mathbf{P} - \mathbf{r}_1|$ . $\psi_1(\mathbf{P}) = A_1 e^{-ikd_1}$
Wave 2 (from slit 2): Propagates a distance $d_2 = |\mathbf{P} - \mathbf{r}_2|$ . $\psi_2(\mathbf{P}) = A_2 e^{-ikd_2}$

(We’ll assume $A_1 \approx A_2 = A$ for simplicity, as the distances are similar).

The total wave at $\mathbf{P}$ is the sum of these two complex numbers:

\psi_{total}(\mathbf{P}) = \psi_1(\mathbf{P}) + \psi_2(\mathbf{P}) = A(e^{-ikd_1} + e^{-ikd_2})

What we actually see (the brightness or intensity) is the magnitude squared of this complex number, $I = |\psi_{total}|^2$ .

\begin{aligned} I(\mathbf{P}) &\propto |A(e^{-ikd_1} + e^{-ikd_2})|^2\\ & = A^2 |e^{-ikd_1} + e^{-ikd_2}|^2 \\ & = A^2 (e^{ikd_1} + e^{ikd_2})(e^{-ikd_1} + e^{-ikd_2}) \\ & = A^2 (2 + e^{ik(d_1 - d_2)} + e^{-ik(d_1 - d_2)}) \\ & = A^2 (2 + 2\cos(k(d_1 - d_2))) \\ & = 4A^2 \cos^2\left(\frac{k(d_1 - d_2)}{2}\right)\end{aligned}

So, this simplifies to:

I(\mathbf{P}) \propto 4A^2 \cos^2\left(\frac{k(d_1 - d_2)}{2}\right)

The intensity depends on the cosine of the difference in path lengths ( $d_1 - d_2$ ).

If the path difference $d_1 - d_2$ is a whole number of wavelengths ( $n\lambda$ ), the phases align ( $k\Delta d = 2\pi n$ ), the waves add up, and we get a bright spot (constructive interference).
If the path difference is a half-wavelength ( $(n+1/2)\lambda$ ), the phases are opposite ( $k\Delta d = \pi(2n+1)$ ), the waves cancel out, and we get a dark spot (destructive interference).

If we assume the slits are separated by a distance $d$ and the screen is far away at distance $L$ , we can approximate the path difference as: $d_1 - d_2 \approx \frac{xd}{L}$
where $x$ is the position on the screen. Plugging this in, we get the classic interference pattern:

I(x) \propto 4A^2 \cos^2\left(\frac{\pi d x}{\lambda L}\right)

Interference is nothing more than the addition of complex phasors, where their relative angles are set by the propagation delay (path length difference).

Diffraction: Interference on a Grand Scale

Diffraction is interference from a continuous number of sources.

Instead of two discrete slits, imagine a single wide aperture (like a single slit, a circular hole, or any shape). The Huygens-Fresnel Principle states that every single point inside that aperture $A$ acts as a new source of waves.

To find the wave $\psi(x, y)$ at a point $\mathbf{P}=(x,y)$ on our screen, we can’t just add two waves. We have to integrate (sum up) the contributions from every point $\mathbf{r}' = (x', y')$ in the aperture.

Let the aperture’s transmission be described by an “input signal” $A(x', y')$ (e.g., $A=1$ inside the slit, $A=0$ outside). The propagation from $(x', y')$ to $(x, y)$ on a screen at distance $L$ introduces a distance $d = \sqrt{L^2 + (x-x')^2 + (y-y')^2}$ .

The total wave at the screen is as follows:

\psi(x, y) = \iint_{aperture} A(x', y') \frac{e^{-ikd}}{d} dx' dy'

This integral is the heart of wave propagation. In signal processing terms:

$A(x', y')$ is the input signal.
$\psi(x, y)$ is the output signal.
$h(x,y; x',y') = \frac{e^{-ikd}}{d}$ is the system’s impulse response (or propagator).

The propagation of the wave is a linear system that “processes” the aperture function.

The Big Reveal: Propagation is the Fourier Transform

If we make a very common approximation—that the screen is far away (the Fraunhofer approximation)—the math simplifies beautifully.

In this far-field limit, the distance $d$ in the phase term can be approximated as:

d \approx L + \frac{(x-x')^2 + (y-y')^2}{2L} \approx L - \frac{xx' + yy'}{L} + \dots

Plugging this into our integral and dropping some constant phase/amplitude factors (which just dim or shift the whole pattern), we get:

\psi(x, y) \propto \iint_{aperture} A(x', y') e^{-i \frac{k}{L} (xx' + yy')} dx' dy'

We can define new spatial frequency coordinates:

$k_x = \frac{kx}{L} = \frac{2\pi x}{\lambda L}$
$k_y = \frac{ky}{L} = \frac{2\pi y}{\lambda L}$

Substituting these in, our diffraction pattern $\psi$ as a function of these new coordinates is:

\begin{aligned} \Psi(k_x, k_y) \propto& \iint A(x', y') e^{-i(k_x x' + k_y y')} dx' dy' = \mathcal{F}\{A(x', y')\}(k_x, k_y)\end{aligned}

This is the 2D Fourier Transform.

This is a profound result:

The wave pattern you see from a far-away aperture (diffraction) is the Fourier Transform of the aperture’s shape.

A wide single slit ( $A(x')$ is a rect function) produces a sinc function pattern ( $\text{sinc}(x) = \sin(x)/x$ ).
A tiny circular hole ( $A(r)$ is a circ function) produces an Airy disk (a Bessel function), which is why stars look like blurry spots, not perfect points, in a telescope.
A double slit (two rect functions) produces the classic interference fringes modulated by a sinc envelope.

A(x') = \text{rect}\left(\frac{x' + d/2}{a}\right) + \text{rect}\left(\frac{x' - d/2}{a}\right) = [\delta(x' + d/2) + \delta(x' - d/2)] * \text{rect}\left(\frac{x'}{a}\right)

The resulting diffraction pattern is:

\begin{aligned}\Psi(k_x) \propto \mathcal{F}^{-1}\{A(x')\} &= \mathcal{F}^{-1}\{[\delta(x' + d/2) + \delta(x' - d/2)] * \text{rect}\left(\frac{x'}{a}\right)\} \\&=\mathcal{F}^{-1}\{\delta(x' + d/2) + \delta(x' - d/2)\} \cdot \mathcal{F}^{-1}\{\text{rect}\left(\frac{x'}{a}\right)\} \\&= 2a \, \text{sinc}\left(\frac{a k_x}{2}\right) \cos\left(\frac{d k_x}{2}\right)\end{aligned}

What it all means:

Propagation = Signal Processing: Wave propagation acts like a physical computer that calculates the Fourier Transform.
Interference: This is just the linear addition of complex signals.
Diffraction: This is the result of a “propagation filter” being applied to an “input aperture signal,” which, in the far-field, is the Fourier Transform.

This math is why signal processing techniques are not just for audio and images, but are fundamental to describing reality itself. The next time you see sunlight filter through a narrow gap, you’re not just seeing light—you’re seeing a Fourier Transform computed by nature.