Differentially Private Gradient Flow based on the Sliced Wasserstein Distance

March 5, 2025

Novel differentially private model using gradient flows defined on an optimal transport metric.

This Research Card introduces a novel, theoretically grounded method for differentially private generative modeling by leveraging a smooth mathematical process, achieving high-fidelity data generation with strong privacy guarantees and lower computational costs compared to traditional approaches.

Title: Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
Short Title: Novel differentially private model using gradient flows defined on an optimal transport metric.
Authors: Ilana SEBAG, Muni Sreenivas Pydi, Jean-Yves Franceschi, Alain Rakotomamonjy, Mike Gartrell, Jamal Atif, Alexandre Allauzen
Team : RSC.FDL. Collaboration with : Miles Team, LAMSADE, Université Paris-Dauphine, PSL University, CNRS and ESPCI PSL.
Status : Published at TMLR (01/2025)
Category : Privacy, Generative Modeling, gradient flows.

Why did we work on this topic (the problem we want to solve)?

Safeguarding data has become essential in this era of widespread AI adoption. Generative modeling, in particular, poses unique challenges due to its ability to learn and replicate intricate data distributions, which risks exposing sensitive information from the original dataset if the model is trained without any privacy component. While existing approaches like adding noise to the gradient (DP-SGD) or using differentially private losses for generator-based methods are effective, they face limitations in balancing three key aspects:

Privacy (How well is the data protected from privacy attacks?),
Fidelity (How realistic and high-quality is the data generated by the model?),
Computational efficiency (How much computation and resources are required to train the model?).

By introducing a novel differentially private algorithm based on gradient flows and the Gaussian-smoothed Sliced Wasserstein Distance, we aim to minimize data leakage while achieving high-fidelity data generation under low privacy budgets and reduced computational costs. This principled alternative addresses unexplored areas in privacy-preserving generative modeling, advancing the field toward more responsible AI development.

How did we proceed?

In this work, we present a novel theoretical framework for a differentially private gradient flow of the sliced Wasserstein distance (SWD), which has not been explored previously as a way to guarantee differential privacy for generative AI models. Our approach involves defining the gradient flow on the smoothed SWD. Although the Gaussian smoothing method appears straightforward, it introduces significant theoretical challenges, particularly regarding the existence and regularity of the gradient flow solution.

To address these complexities, we establish the “continuity equation” for our new gradient flow of the smoothed SWD, resulting in a smoothed velocity field that governs how the generated data is privately produced. This allows us to discretize the continuity equation from the previous step, into a Stochastic Differential Equation (SDE) that ensures the gradient flow maintains differential privacy. Notably, we show that after discretization, the smoothing process in the drift term functions as a Gaussian mechanism, ensuring that the privacy budget is carefully tracked throughout the process.

What is the originality here? / What did we achieve?

On the theoretical front, our contribution is significant as we prove, for the first time in the literature, the existence and regularity of the gradient flow for the Gaussian-smoothed Sliced Wasserstein distance (GSW). The proof strategies we use, inspired by previous works, require extensive modification to handle the unique characteristics of the GSW. This novel theoretical result lays the foundation for future work on differential privacy gradient flows, opening the door to new possibilities and improvements in privacy preserving AI.

From an experimental standpoint, we show that our proposed approach outperforms the baseline DPSWgen model, which uses a generator-based architecture with the differentially private Sliced Wasserstein loss, across various privacy budgets (levels of privacy). Our method not only achieves better FID scores but also generates higher-quality samples, demonstrating the practical viability and superior performance of our approach in safeguarding privacy while producing high-fidelity generative models.