Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

Although machine learning models trained on massive data have led to\nbreak-throughs in several areas, their deployment in privacy-sensitive domains\nremains limited due to restricted access to data. Generative models trained\nwith privacy constraints on private data can sidestep this challenge, providing\nindirect access to private data instead. We propose DP-Sinkhorn, a novel\noptimal transport-based generative method for learning data distributions from\nprivate data with differential privacy. DP-Sinkhorn minimizes the Sinkhorn\ndivergence, a computationally efficient approximation to the exact optimal\ntransport distance, between the model and data in a differentially private\nmanner and uses a novel technique for control-ling the bias-variance trade-off\nof gradient estimates. Unlike existing approaches for training differentially\nprivate generative models, which are mostly based on generative adversarial\nnetworks, we do not rely on adversarial objectives, which are notoriously\ndifficult to optimize, especially in the presence of noise imposed by privacy\nconstraints. Hence, DP-Sinkhorn is easy to train and deploy. Experimentally, we\nimprove upon the state-of-the-art on multiple image modeling benchmarks and\nshow differentially private synthesis of informative RGB images. Project\npage:https://nv-tlabs.github.io/DP-Sinkhorn.\n