Research Card: Unifying GANs and Score-Based Diffusion as Generative Particle Models

March 5, 2024

How to merge two different types of generative models into a single framework, allowing the discovery of new types of generative models via hybridization.

Paper: Unifying GANs and Score-Based Diffusion as Generative Particle Models
Authors: Jean-Yves Franceschi (Criteo AI Lab), Mike Gartrell (*Criteo AI Lab), Ludovic Dos Santos (Criteo AI Lab), Thibaut Issenhuth (Criteo AI Lab & Ecole des Ponts), Emmanuel de Bézenac (ETH Zürich), Mickaël Chen (http://Valeo.ai), Alain Rakotomamonjy (Criteo AI Lab).
Category: Generative AI, Deep Learning
Revue: NeurIPS 2023

Video Summary — 1 minute

What is the problem we want to solve?

Generative AI has emerged as a groundbreaking approach for generating realistic data (text, images, videos, audio, etc.). For non-textual data, two prominent models have made waves: diffusion models (for instance, DALL-E 2, Midjourney, Stable Diffusion) and Generative Adversarial Networks (GANs).

Diffusion models rely on a simple principle: if noising destroys data, then denoising generates data. Indeed, through successive addition of noise to the data (the noising process), one can destroy the data by transforming it into pure noise. The generative process of diffusion models performs the opposite: they learn to recover the data by reversing this destructive noising process (the generating denoising process) with a neural network.

Principle of a diffusion model. (Top) Noising process. (Bottom) Learned denoising process.

Example of generation (progressive denoising) from a diffusion model.

This is in contrast with GANs, the most predominant technique of the last decade, now out of fashion, that relies on a more complex but clever principle. GANs use two neural networks, a generator, and a discriminator, engaged during training in an adversarial back and forth. The generator is tasked to produce realistic samples, while the discriminator’s role is to discern if the sample is real or fake. After training, only the generator is used for generation. Unlike diffusion, the generation is not progressive, hence very fast.

It seems that GANs and diffusion models are then fundamentally different, which explains the apparent superiority of Diffusion over GANs in terms of generative performance. This interpretation is widely accepted by the community but has never been challenged. In this work, we tackle this problem and investigate the differences between both models in unifying them in a novel framework.

How did we go about it? What did we find?

We carefully studied the underlying equations driving the behavior of GANs and diffusion. We found a strong link between both models: the denoising process in diffusion actually relies on equations that are very similar to those governing the training of GANs. From there, we were able to formalize them in a single mathematical framework. Using this new unified framework, we introduced two new types of generative models that are hybridizations of GANs and diffusion: a diffusion model with a generator, and a GAN without generator. The obtained promising results pave the way for future improvements of generative models using our framework.

What’s original about it?

The unveiled link between GANs and diffusion is a counterintuitive result for the generative AI community as these models were considered as inherently different from each other. Furthermore, the introduce hybrid models are original in their design as we are the first to add/remove a generator in diffusion / GANs.