Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning

It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from\nworse generalization performance than SGD despite their faster training speed.\nThis work aims to provide understandings on this generalization gap by\nanalyzing their local convergence behaviors. Specifically, we observe the heavy\ntails of gradient noise in these algorithms. This motivates us to analyze these\nalgorithms through their Levy-driven stochastic differential equations (SDEs)\nbecause of the similar convergence behaviors of an algorithm and its SDE. Then\nwe establish the escaping time of these SDEs from a local basin. The result\nshows that (1) the escaping time of both SGD and ADAM~depends on the Radon\nmeasure of the basin positively and the heaviness of gradient noise negatively;\n(2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly\nbecause (a) the geometry adaptation in ADAM~via adaptively scaling each\ngradient coordinate well diminishes the anisotropic structure in gradient noise\nand results in larger Radon measure of a basin; (b) the exponential gradient\naverage in ADAM~smooths its gradient and leads to lighter gradient noise tails\nthan SGD. So SGD is more locally unstable than ADAM~at sharp minima defined as\nthe minima whose local basins have small Radon measure, and can better escape\nfrom them to flatter ones with larger Radon measure. As flat minima here which\noften refer to the minima at flat or asymmetric basins/valleys often generalize\nbetter than sharp ones , our result explains the better generalization\nperformance of SGD over ADAM. Finally, experimental results confirm our\nheavy-tailed gradient noise assumption and theoretical affirmation.\n