Why Does Sharpness-Aware Minimization Generalize Better Than SGD?

The challenge of overfitting, in which the model memorizes the training data and fails to generalize to test data, has become increasingly significant in the training of large neural networks. To tackle this challenge, Sharpness-Aware Minimization (SAM) has emerged as a promising training method, which can improve the generalization of neural networks even in the presence of label noise. However, a deep understanding of how SAM works, especially in the setting of nonlinear neural networks and classification tasks, remains largely missing. This paper fills this gap by demonstrating why SAM generalizes better than Stochastic Gradient Descent (SGD) for a certain data model and two-layer convolutional ReLU networks. The loss landscape of our studied problem is nonsmooth, thus current explanations for the success of SAM based on the Hessian information are insufficient. Our result explains the benefits of SAM, particularly its ability to prevent noise learning in the early stages, thereby facilitating more effective learning of features. Experiments on both synthetic and real data corroborate our theory.

Paper

References (37)

Scroll for more · 25 remaining

Similar papers

Peer review

Reviewer EhwP7/10 · confidence 4/52023-06-16

Summary

This paper presents an in-depth theoretical examination of Sharpness-Aware Minimization (SAM) in the context of feature learning. The authors identify the challenge of overfitting in the training of these networks, a problem that becomes more prominent as the model size increases. The traditional gradient-based methods like Gradient Descent (GD) and Stochastic Gradient Descent (SGD) are identified to suffer from unstable training and harmful overfitting. The authors highlight SAM as a promising alternative that has demonstrated improved generalization even in situations where label noise is present. However, they point out the current lack of understanding on why SAM outperforms SGD, especially in nonlinear neural networks and classification tasks. The core contribution of this paper is filling this knowledge gap by providing theoretical reasoning on why SAM has better generalization than SGD, particularly in the context of two-layer convolutional ReLU networks. The authors characterize the conditions under which benign overfitting can occur when using SGD, and further demonstrate a phase transition phenomenon unique to SGD. They formally prove that under these same conditions, SAM can achieve benign overfitting, and therefore has a distinct advantage in terms of generalization error. Specifically, the paper highlights the ability of SAM to mitigate noise learning, which prevents it from succumbing to harmful overfitting early on in the training process. This aspect of SAM allows it to facilitate more effective learning of weak features. The theoretical findings are supported by experiments on both synthetic and real data, bolstering the credibility of the presented theory.

Strengths

- **Addressing a Significant Issue**: The authors tackle the important issue of overfitting in the training of large neural networks. They delve into a key challenge facing the field of deep learning, thus making their work relevant and timely. - **Theoretical Contributions**: This paper provides a strong theoretical analysis of why Sharpness-Aware Minimization (SAM) outperforms Stochastic Gradient Descent (SGD), particularly for two-layer convolutional ReLU networks. This contributes to the current understanding of how to improve the generalization of large neural networks. - **Comprehensive Study**: The authors carry out an in-depth comparison of SAM and SGD, using both synthetic and real data. This robust approach adds to the validity and comprehensiveness of their findings. - **Novelty**: The authors claim that this is the first benign overfitting result for a neural network trained with mini-batch SGD. This claim, if validated, could signify a novel contribution to the field. - **Clarity and Organization**: The paper appears to be well-structured and clearly written, with a good overview of the background literature, a clear statement of the problem, and a detailed account of the authors' contributions.

Weaknesses

- **Limited Scope**: The paper's focus on two-layer convolutional ReLU networks with fixed second layer might limit the generalizability of its findings to other types of networks. Further research could be needed to determine if SAM has similar benefits for different architectures. - **Presentation Could Be Enhanced**: The current version of the paper could benefit from further polishing in terms of presentation. For instance, the y-axis in Figure 2 lacks clarity — it's not immediately evident whether it represents a normal scale or a logarithmic scale. Also, the 'dimension' it not clear. The term 'clear samples' could be more accurately represented as 'clean samples.' Additionally, there's inconsistency in the usage of 'P-1' and 'P', which should ideally be standardized throughout the paper for improved readability and comprehension. - **Partial Theoretical Results**: The authors provide a thorough analysis of both benign and harmful overfitting scenarios for SGD. However, the corresponding harmful overfitting regime for SAM appears to be absent from the theoretical results, which could make the comparison somewhat incomplete.

Questions

Could you please share your thoughts on the potential behavior of SAM within the gray region depicted in Figure 1?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

I believe the authors could provide a more extensive evaluation of potential limitations inherent in their research. Engaging in a comprehensive discussion regarding these constraints would strengthen the overall rigor and transparency of the study.

Reviewer CEVG8/10 · confidence 4/52023-06-29

Summary

This paper studies the question of why SAM generalizes better than SGD on a specific binary classification task. The task looks like a special case of the sparse coding model where the relevant parameter is a signal-to-noise ratio (SNR). The authors present theoretical results on the performance of SAM and SGD for this specific task. More specifically, their main results say that SGD requires higher SNR in the data distribution to generalize, while SAM can generalize even with smaller SNR.

Strengths

- The paper has a very comprehensive end-to-end analysis for neural network learning, which is usually very technically complicated. - The paper is overall well-written and it was easy to follow. - Figure 1 and Figure 2 have very nice resemblance.

Weaknesses

- Discussion of the results seems not quite comprehensive; see the comments below.

Questions

- The binary classification considered in this work seems to be an instantiation of the well known sparse coding problem. It would be helpful to motivate the model by presenting connection to the well known sparse coding problem. - Connection to large learning rates GD/SGD. Recently there have been several works trying to understand why large learning rate GD/SGD generalize better than small learning rate ones. It would be clarifying to see if that's also the case for the model considered in this work. It seems that there are recent works that show that large learning rates help with generalization for a sparse coding problem (e.g. [1]). Given this, it would be a nice clarification to test at least empirically if one can achieve SAM's performance with larger learning rate GD/SGD. - Related to the comment about the large learning rate, Q. Is the result for SGD also true for Gradient Flow (continuous dynamics of SGD)? - The theoretical results show that SAM updates help prevent memorizing the spurious feature (or harmful overfitting) during the early stage of the training. For the model in this paper, does that mean you can switch from SAM to SGD after a few epochs and basically achieve the same performance as SAM? - The required perturbation radius ($\tau$) in theory scales inversely with $\sqrt{d}$, which seems much smaller than what people choose in practice. Does this requirement consistent with the experiments in the paper? I think it's an important point since a sufficiently large $\tau$ distinguishes SAM from SGD. - The original motivation for SAM was to encourage the landscape around the solution to be more flat. Does that intuition also hold true for the sparse coding problem considered in this work? Does the prevention of harmful memorization lead to a flatter landscape when you consider the flatness around the iterates of SAM? - Does the oscillation of SAM (dynamics characterized in the previous works mentioned in this paper) also occur in the training dynamics of SAM for the model in this paper? If so, does the oscillation another explanation as to why SAM does not learn spurious features? - In the experiments, it seems that only full-batch GD/SAM are tested for the synthetic model setting. Could you comment on the behavior for SGD and Stochastic SAM? - Also, many recent works have studied un-normalized variant of SAM (usually referred to as USAM), where the ascent step normalization is removed. Are the main results for SAM also true for USAM or is the normalization necessary for the main results? - In the related work section, it would be nice to see discussion on other techniques (large learning rates, label noise etc) as there have been many works on their effects on generalization (e.g. [1] [2] [3]). - [1] Learning threshold neurons via the "edge of stability" (https://arxiv.org/abs/2212.07469) - [2] SGD with Large Step Sizes Learns Sparse Features (https://arxiv.org/abs/2210.05337) - [3] What Happens after SGD Reaches Zero Loss? --A Mathematical Framework (https://arxiv.org/abs/2110.06914) - [4] Label Noise SGD Provably Prefers Flat Global Minimizers (https://arxiv.org/abs/2106.06530)

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Please see the comments in the Question section.

Reviewer jpeN6/10 · confidence 4/52023-07-05

Summary

The paper aims to provide a theoretical basis for the superiority of SAM over SGD. Different from former explanation based on Hessian information, the authors firstly discuss the loss landscape of non-smooth neural networks like two-layer convolutional ReLU networks. Notably, the paper proves that under conditions of harmful overfitting using SGD, SAM can deliver benign overfitting and outperform SGD in terms of generalization error. The experiments are conducted to demonstrate that SAM can outperforms SGD in terms of generalization error by mitigating noise learning and enhancing the efficiency of weak feature learning.

Strengths

- The motivation is clear and novel. Previous researches focused on shallow model with implicit smooth loss. As a more challenging setting, the paper studies two-layer convolutional networks with ReLU units. - The theoretical analysis is convincing. In terms of benign overfitting, the authors discuss the phenomenon and condition in SGD firstly. In harmful overfitting, the authors theoretically prove that SAM can outperform SGD in the early stage of learning.

Weaknesses

- The experiment results may not be reliable. In Section 5, the author sets the learning rate to 0.1. However, according to recent research [1] and my practical experience, I suspect the learning rate is too high and may lead the model to overfit easily. The model may not easily overfit if we use SGD with suitable learning rate. Hence, it might be better to add an experiment and discuss the influence of learning rate. - Potential overclaiming based on the analysis and experiment. At the beginning of the paper, the authors claim that SAM can help model learn weak features effectively. However, I do not see obvious related content that can support the authors claim. I recommend the authors modify their claims or try to discuss the relationship between SAM and weak features more clearly. [1] Andriushchenko et al. “A Modern Look at the Relationship between Sharpness and Generalization”, ICML, 2023. ------- After reading the responses from the authors, my concerns has been well addressed. I will keep my score as weak accept.

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reviewer TY4G5/10 · confidence 3/52023-07-05

Summary

This paper presents two theoretical contributions regarding benign/malign overfitting of two-layer convolutional ReLU neural networks. For an idealized data distribution, it i) gives the conditions (with respect to the dimension of the data and to the signal strength) under which benign/malign overfitting occurs when training the network with mini-batch-SGD; ii) shows that training the network with SAM leads benign overfitting more often than SGD and in particular in many situations where SGD suffers from malign overfitting. These findings are illustrated by a numerical experiment conducted in the idealized situation used for the theoretical analysis.

Strengths

The two contributions presented in this paper are significant because Contribution i) improves on the literature, in particular with respect to the closest work (Kou et al., 2023), which proposes exactly the same analyses but for a gradient descent training; Contribution ii) is, up to my knowledge, the first such result for SAM. The latter result provides a comprehension on the well-known fact that SAM helps networks to generalize better than SGD. This is a real breakthrough, which should be welcomed by the community.

Weaknesses

I was very enthusiastic by the results presented in this paper but I became disillusioned when reading it. The presentation is disordered and in particular, notations are not all defined, making the results (even informally stated) very difficult to understand. After having a glance at the appendix (which seems to be rigorously written), it seems that the manuscript is an assembly of results extracted from the appendix, unfortunately awkwardly built. As for me, it is a pity because I think that the results are important and of interest for the community, but the presentation is inadequate and not clear enough. Comments: 1) The abstract misses to state Contribution i) (which constitutes a large part of the manuscript and which is clearly reminded in the conclusion). 2) Definition 2.1 is difficult to understand. In particular, the parts “signal contained in each data point” and “the other are given by” are unclear. Besides, even though this model seems common in the recent literature, a discussion regarding its limitations, for instance independence between labels and covariates, would be appreciated. The same remark can be done regarding the architecture of the neural network considered (Section 2.2). In addition, Condition 3.1 is discussed only quickly Line 142 ; a deeper discussion regarding the role of the variances and the polynomial degrees appearing would be appreciated. 3) There is no link between statements/results in the paper and their counterparts in the appendix. Since the reader is supposed to juggle the two, it would be helpful to know where are the formal statements and the proofs in the appendix. 4) I understand that authors try to explain the derivation of their main results but I am not very comfortable with informal statements, first because of the notation problem previously stated and also because some conclusions are given with vague explanations (in my case they are difficult to grasp) while they seem to be quite difficult to obtain formally. This is the case for instance Line 218 regarding the symmetry of $\rho$. 5) Line 117, the authors invoke the discontinuity of the gradient to justify the need of an analysis based on an other technique than the Taylor expansion. As for me an even better argument is that the ReLU function is formally not differentiable everywhere. 6) The statement of Theorem 4.1 is a bit unclear: the parts “we train/we can train” could be replaced by passive forms, “neural networks” refers, as far as I understand, to the chosen architecture. In addition, mentioning SGD in a result concerning SAM may throw the reader. It could be specified beforehand that SAM training uses intrinsically SGD (but at a point which is not the current iterate). 7) Lemma 4.3 is true only for a particular choice of $\tau$, as stated in Theorem 4.1 (I am not sure that this is clear in Lemma C.5). Since this result is important, this should appear in Lemma 4.3 and discussed after. 8) The related work section (Section 6) appears at the end of the manuscript but is an enumeration of papers. Such an enumeration is generally well placed just after the introduction. Placing the related work section after the result statements makes sense if it discusses technical differences with the closest papers. As it happens, this work seems to be based to a certain extent on (Kou et al., 2023). Thus, it would be enriching to discuss the contribution, and particularly the technical novelties (GD → SGD/SAM), of this work with respect to (Kou et al., 2023). 9) Mathematical remarks: Line 47, $t$, $\mathbf W^{(t)}$, $\boldsymbol \mu$, $L_{\mathcal D}^{0-1}$, $p$, $\Omega$ are not defined; Lines 47, 150, 240 and after, the expression “converges to $\epsilon$” should be replaced by “converges to $0$”; Line 66, $l_2$ → $\ell_2$, Line 68, absolute values in $|a_k/b_k|$ seem useless; “omit logarithmic terms” should be defined explicitly; Line 97, $\mathbf W$ is not defined; Line 100, “is a collection of” means “is a matrix”, doesn’t it? Line 101, $[n]$ is not defined; Line 113, it is specified that $\sigma_0^2$ is the variance of the normal distribution but not Line 80; in Equation (5), the 2-norm should be a Frobienus norm; what is the utility of Equation (6) with respect to the equation Line 175? Line 121, $S_i^{(t, b)}$ → $\tilde S_i^{(t, b)}$; Line 219, $T^*$ is not defined; Lines 218 and 226, the authors could remind what are $H$ and $B$; Line 226, I understand that $|\cdot|$ is the cardinality but this is not stated. 10) Typographical remarks: Line 9, “for the certain” → “for a certain”; Line 27, “with minimal gradient” → “with minimal gradient norm”; Figure 1, blue and yellow are inverted; Line 102, “cross-entropy loss function” and “logistic loss” are redundant; Line 115, “indicator function” → “the indicator function”; Line 156, “Bayesian optimal risk” → “Bayes risk”; full points are missing Lines 202, 207, 214; Line 237: “iteration” → “iterations”; Line 251, full point instead of colon; Figure 2, y-label is cut; Line 299, “an generalization” → “a generalization”.

Questions

1) Equations Line 175 and 176 are given as a definition (Definition 3.3), but appear to be a result of the data distribution and of the network architecture chosen. Why are they presented as a definition? 2) The numerical experiment is performed with gradient descent instead of SGD, while the theoretical analysis deeply relies on SGD. Why? Are results identical with SGD? 3) Similarly, are the theoretical results for SAM the same if optimization is performed with gradient descent?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

4 excellent

Limitations

See above regarding mathematical assumptions. Societal impact is not addressed.

Reviewer TY4G2023-08-11

I would like to thank the authors for their rebuttal, which I read carefully. I acknowledge that authors accept to change the paper presentation according to my comments and I agree to increase my score consequently. Since I do not have access to the revised version, the score is increased of one level.

Authorsrebuttal2023-08-11

Thank you!

Thank you for raising the score and for your constructive comments! We will be sure to incorporate your suggested changes and revise our paper accordingly in the final version.

Reviewer CEVG2023-08-13

Thank you for your response

I read the author response, and it's excellent. Assuming that the authors will reflect all of them in the final version, I raise my score to 8. Great work and congrats! Also, regarding Q9 "Are the main results for SAM also true for USAM or is the normalization necessary for the main results?," if the authors have some intelligent things to say regarding that question, please add the results/discussions in the final version. It would be very helpful for the readers in light of the following recent work: Dai et al. 2023 (See the reference below.) In a nutshell, this recent work seems to show that for SAM's practicality, the normalization part is **necessary**. Since the theoretical results in this work show that the normalization step is necessary for the generalization of the resulting solution, this would give a lot of intuitions for practitioners and valuable insights for the ML community! Dai et al. "The Crucial Role of Normalization in Sharpness-Aware Minimization" (https://arxiv.org/abs/2305.15287)

Authorsrebuttal2023-08-13

Thank you!

Thank you for raising the score and for your positive feedback! We will make sure to incorporate all the promised changes into our final version. Thank you for pointing out the additional related work on the necessity of normalization in SAM. We will discuss this work as well and articulate our insights in the final version. Indeed, understanding the role of normalization is an important problem that may gain some insights from our theoretical analysis.

Reviewer EhwP2023-08-14

Thank you for your rebuttal to my comments and questions. I appreciate the time and effort you have put into addressing each of my concerns. As a final step in my review, I would like to request access to the code of the experiments you used during the rebuttal. This will allow me to further verify the results and ensure the integrity of the conclusions drawn. Once again, thank you for your detailed responses. I am satisfied with the reponses you have proposed, and I believe they will enhance the quality and clarity of the paper.

Authorsrebuttal2023-08-14

Thank you very much for your positive feedback! In compliance with the rebuttal and discussion guideline, we have made the code available at an anonymous link and sent it to the Area Chair. We anticipate that you will get access to the code very soon. Once again, we really appreciate your valuable comments and suggestions, which help us improve our work!

Reviewer jpeN2023-08-21

Thank you for the detailed response. My concerns have been well addressed, so I will keep my score as weak accept.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC