Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?

While additional training data improves the robustness of deep neural\nnetworks against adversarial examples, it presents the challenge of curating a\nlarge number of specific real-world samples. We circumvent this challenge by\nusing additional data from proxy distributions learned by advanced generative\nmodels. We first seek to formally understand the transfer of robustness from\nclassifiers trained on proxy distributions to the real data distribution. We\nprove that the difference between the robustness of a classifier on the two\ndistributions is upper bounded by the conditional Wasserstein distance between\nthem. Next we use proxy distributions to significantly improve the performance\nof adversarial training on five different datasets. For example, we improve\nrobust accuracy by up to 7.5% and 6.7% in $\\ell_{\\infty}$ and $\\ell_2$ threat\nmodel over baselines that are not using proxy distributions on the CIFAR-10\ndataset. We also improve certified robust accuracy by 7.6% on the CIFAR-10\ndataset. We further demonstrate that different generative models bring a\ndisparate improvement in the performance in robust training. We propose a\nrobust discrimination approach to characterize the impact of individual\ngenerative models and further provide a deeper understanding of why current\nstate-of-the-art in diffusion-based generative models are a better choice for\nproxy distribution than generative adversarial networks.\n