The Maths Behind Generative AI: GANs, VAEs and Diffusion Models

A generative model's job, stripped of the architecture diagrams, is always the same underlying statistics problem: learn the probability distribution that real data was drawn from, well enough that you can draw a new sample from it that looks like it could have come from the same place. GANs, VAEs and diffusion models all solve that one problem — they just take completely different mathematical routes to get there.

GANs: turning generation into a contest

A Generative Adversarial Network sidesteps the probability distribution question almost entirely, by turning generation into a two-player game. A generator network turns random noise into a fake sample; a discriminator network is trained to tell real samples from fake ones. The generator is trained to fool the discriminator; the discriminator is trained not to be fooled. Formally, it's a minimax game over a value function, and the theoretical fixed point — where the generator perfectly matches the real data distribution and the discriminator can only guess at 50/50 — is a Nash equilibrium.

In practice that equilibrium is hard to reach reliably. Training can oscillate rather than converge, and a well-known failure mode called mode collapse happens when the generator finds a small handful of outputs that reliably fool the current discriminator and stops exploring further — technically fooling the game, while badly under-representing the real distribution's actual diversity.

VAEs: an honest probability model, approximated

A Variational Autoencoder takes the more direct route: it assumes each data point is generated from some hidden latent variable z, and tries to learn that generative process properly. The catch is that computing the true posterior distribution over z given an observed data point is intractable for anything but the simplest models. A VAE's answer is to train a second network to approximate that posterior, and instead of maximising the data likelihood directly (which it can't compute), it maximises a provable lower bound on it — the Evidence Lower Bound, or ELBO — which splits cleanly into a reconstruction term and a KL-divergence term that keeps the learned latent space close to a simple, well-behaved prior.

That KL term is what makes the latent space genuinely useful: it keeps similar things near each other and stops the model from just memorising isolated points, which is why VAE latent spaces support smooth interpolation between two very different inputs, one of the method's most useful properties.

Diffusion models: generation as controlled undoing

A diffusion model takes a third, very different route. The forward process is fixed and requires no training at all: take a real image and add a small amount of Gaussian noise, many times over, until nothing recognisable is left. The only thing that actually gets learned is the reverse — a network trained to predict, at each step, what noise was just added, so it can be subtracted back out. Run that learned reverse process starting from pure random noise, and you generate a brand new sample.

This connects directly back to stochastic calculus: the forward noising process is a stochastic differential equation, and the network is effectively learning the score function needed to run that equation's reverse-time version. It's a much more stable training target than a GAN's adversarial game — no two networks fighting each other, just one network learning to predict noise — which is the main reason diffusion models became the dominant approach for image generation.

The pattern worth remembering across all three: a GAN avoids modelling the distribution directly and uses a contest instead; a VAE models it honestly but has to approximate an intractable step; a diffusion model turns generation into many small, easy denoising predictions instead of one hard one.

What's still genuinely unsolved

Three questions in this area don't have settled answers yet, even at research level. Evaluation is harder than it looks — there's no single number that reliably captures whether a generated sample is actually good; standard metrics like FID compare statistics of learned feature representations, not genuine human judgement, and are known to be gameable. Controllability — reliably steering what a model generates via a prompt or a conditioning signal, rather than broadly influencing it — remains an active research area rather than a solved engineering problem. And the deepest open question is about theoretical limits: whether these models are capable of genuine generalisation beyond their training distribution, or whether what looks like novelty is sophisticated interpolation within it — a question about the models' fundamental capabilities, not just their current implementation.

None of that is a reason to treat the field as unrigorous — quite the opposite. Knowing exactly where the open questions are is what separates a genuine understanding of generative AI from being able to use the tools without knowing how they work. If postgraduate study of AI's mathematical foundations is where you or your student is headed, that's exactly the depth we teach at — see our postgraduate AI tutoring or the full learning pathway here.