Not all noise is accounted equally: How differentially private learning benefits from large sampling rates

Learning often involves sensitive data and as such, privacy preserving\nextensions to Stochastic Gradient Descent (SGD) and other machine learning\nalgorithms have been developed using the definitions of Differential Privacy\n(DP). In differentially private SGD, the gradients computed at each training\niteration are subject to two different types of noise. Firstly, inherent\nsampling noise arising from the use of minibatches. Secondly, additive Gaussian\nnoise from the underlying mechanisms that introduce privacy. In this study, we\nshow that these two types of noise are equivalent in their effect on the\nutility of private neural networks, however they are not accounted for equally\nin the privacy budget. Given this observation, we propose a training paradigm\nthat shifts the proportions of noise towards less inherent and more additive\nnoise, such that more of the overall noise can be accounted for in the privacy\nbudget. With this paradigm, we are able to improve on the state-of-the-art in\nthe privacy/utility tradeoff of private end-to-end CNNs.\n