ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks

Recently, learning algorithms motivated from sharpness of loss surface as an\neffective measure of generalization gap have shown state-of-the-art\nperformances. Nevertheless, sharpness defined in a rigid region with a fixed\nradius, has a drawback in sensitivity to parameter re-scaling which leaves the\nloss unaffected, leading to weakening of the connection between sharpness and\ngeneralization gap. In this paper, we introduce the concept of adaptive\nsharpness which is scale-invariant and propose the corresponding generalization\nbound. We suggest a novel learning method, adaptive sharpness-aware\nminimization (ASAM), utilizing the proposed generalization bound. Experimental\nresults in various benchmark datasets show that ASAM contributes to significant\nimprovement of model generalization performance.\n