Large-batch stochastic gradient descent (SGD) is widely used for training in\ndistributed deep learning because of its training-time efficiency, however,\nextremely large-batch SGD leads to poor generalization and easily converges to\nsharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from\nconverging to good minima. To overcome this difficulty, we propose gradient\nnoise convolution (GNC), which effectively smooths sharper minima of the loss\nfunction. For DP-SGD, GNC utilizes so-called gradient noise, which is induced\nby stochastic gradient variation and convolved to the loss function as a\nsmoothing effect. GNC computation can be performed by simply computing the\nstochastic gradient on each parallel worker and merging them, and is therefore\nextremely easy to implement. Due to convolving with the gradient noise, which\ntends to spread along a sharper direction of the loss function, GNC can\neffectively smooth sharp minima and achieve better generalization, whereas\nisotropic random noise cannot. We empirically show this effect by comparing GNC\nwith isotropic random noise, and show that it achieves state-of-the-art\ngeneralization performance for large-scale deep neural network optimization.\n