Fundamental Convergence Analysis of Sharpness-Aware Minimization

The paper investigates the fundamental convergence properties of Sharpness-Aware Minimization (SAM), a recently proposed gradient-based optimization method [Foret et al., 2021] that significantly improves the generalization of deep neural networks. The convergence properties, including the stationarity of accumulation points, the convergence of the sequence of gradients to the origin, the sequence of function values to the optimal value, and the sequence of iterates to the optimal solution, are established for the method. The universality of the provided convergence analysis, based on inexact gradient descent frameworks Khanh et al. [2023b], allows its extensions to efficient normalized versions of SAM such as F-SAM [Li et al., 2024], VaSSO [Li and Giannakis, 2023], RSAM [Liu et al., 2022], and to the unnormalized versions of SAM such as USAM [Andriushchenko and Flammarion, 2022]. Numerical experiments are conducted on classification tasks using deep learning models to confirm the practical aspects of our analysis.

Paper

Similar papers

Peer review

Reviewer cPcD6/10 · confidence 3/52024-06-26

Summary

The authors provide a fundamental convergence analysis of sharpness-aware minimization algorithms, including the normalized and unnormalized variants. The analysis of normalized variants is based on the theoretical results of inexact gradient descent methods, while the analysis of unnormalized variants is based on the inexact gradient descent method with relative errors. Fundamental convergence properties have been established for SAM algorithms in both convex and non-convex cases.

Strengths

- SAM is an optimizer widely used in searching for flat minima, thus, studying its convergence is an important problem. - The paper establishes theoretical convergence for both normalized and unnormalized SAM algorithms, while most existing convergence results are only for unnormalized variants. As the normalized SAM is more standard than the unnormalized SAM, establishing convergence for normalized SAM is a key contribution. - Compared with existing analysis on SAM, many new fundamental properties are established in Sec 3.2 and Sec.4 - the paper is well-written, easy to follow

Weaknesses

- I think the analysis of the convex case (Sec 3.1) is meaningless. As for the convex function, usually, we can use some convex programming methods to obtain the global optimal solution. - If Theorem 3.3 and Theorem 3.4 are from existing works, some citations for them are necessary (e.g., [Khanh et al., 2023a,b,c, 2024]) - In Corollary 3.5, the assumption on $\rho_t$ is not standard. In practice, $\rho_t$ is a constant, e.g., 0.05 - The convergence analysis of SAM in this paper cannot provide some theoretical insights on the advantage of SAM over ERM.

Questions

- the results reported in Table 4 are much lower than other publications. e.g., - resnet18, cifar10, 94.10 vs 96.52 (https://arxiv.org/pdf/2110.03141) - resnet18, cifar100, 71.77 vs 80.17 (https://arxiv.org/pdf/2110.03141)

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

n/a

Reviewer ZtLt4/10 · confidence 4/52024-07-06

Summary

This paper studies the convergence properties of Sharpness-Aware Minimization (SAM) and some of its variants. The authors use the inexact gradient descent to present convergence guarantees for convex and non-convex cases.

Strengths

I think it is interesting to unify different variants of SAM under the inexact gradient framework.

Weaknesses

- The results do not say much about the quality of the solution in the non-convex case. I think this makes the results less interesting in practice. - The results seem rather a straightforward application of prior work. I think the contribution of the work is limited. - The results do not discuss the stochastic setting. I think based on these points, the results are rather limited and I don't think the paper comes up with a new proof technique.

Questions

NA

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

NA

Reviewer Yksm6/10 · confidence 4/52024-07-09

Summary

In this paper, the authors provide a comprehensive understanding of the SAM and its variants. They establish the convergence for SAM under different settings. Unlike the convergence results in the literature, they establish the last-iterate convergence rather than the best-iterate convergence or function value convergence for SAM. By regarding SAM as an inexact gradient descent method (IGD), their analysis can also be applied to other variants.

Strengths

* They establish the fundamental convergence properties for SAM and its variants. * Their analysis is universal and can be applied to other versions of SAM. * The paper is well-written and easy to follow.

Weaknesses

* In line 480, you should refer to Proposition 3.9 in Khanh et al. * Without the KL assumption, would it be possible to derive the convergence rates for SAM based on your analysis? * The main idea is to regard the SAM as a special case of IGD. Would you please elaborate more on your theoretical contribution compared with IGD?

Questions

See the Weakness part.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

They have mentioned some in their paper.

Reviewer wdGg6/10 · confidence 5/52024-07-24

Summary

### Summary: This paper studies various notions of convergence for SAM. The results are summarized in Table 3. Indeed, in Theorem 3.2, they prove the convergence of SAM for smooth convex function for several definitions under diminishing stepsize. Moreover, in Figure 1, they argue that SAM cannot converge with constant stepsize. In Theorem 3.3 they use Inexact Gradient Descent (IGD) in Algorithm 1 to prove convergence results in the non-convex setting, and they generalize it to unnormalized gradient methods like U-SAM in Theorem 4.1. The paper concludes with experiments.

Strengths

### Pros: - study of convergence of SAM is an important problem in optimization - well citation to previous related works

Weaknesses

### Cons: - the paper is not well-written - the method for non-convex convergence (IGD) is not well motivated and it seems too general to be used for the study of SAM

Questions

### Questions/Comments: This is an interesting paper about the theoretical understanding of SAM convergence. However, I think this paper should be majorly revised. The authors introduce a lot of notions and formulae in the first pages and later without accurately explaining them in the paper. I think because this paper could have general audience, it should be understandable to most part of the community. I recommend authors to move a lot of detailed formulae in the main body of the paper and explain necessary definitions/ideas in words through the paper. The writing can be improved for the next version of the paper. Moreover, the use of IGD seems too general for proving SAM convergence. The authors fail to well motivate using IGDs, as well as how they are enough for the study of SAM. In other words, SAM is way more structured than IGDs and the considered framework seems too general. More explanations are expected for this. - line 42 -- typo

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Reviewer wdGg2024-08-12

Dear authors, thank you for your response. I believe your work is indeed interesting and influential, and my concerns are mainly about its presentation. The use of IGD is less motivated, and the paper is less accessible to a broad audience at this conference. However, to acknowledge the value of your work, I have decided to slightly increase my initial score. I strongly recommend making significant revisions so that the message of the paper is clearer to a broader audience.

Authorsrebuttal2024-08-13

Dear Reviewer wdGg, Thank you very much for reexamining our paper. We will make revisions to the paper following your suggestions. Best regards, Authors

Reviewer cPcD2024-08-12

reply to the rebuttal

I appreciate the authors' response. The first three concerns are resolved. As this is a theoretical paper, I think the 4th concern about experiments is not very crucial. Hence, I will raise my score to 6 to reflect it.

Reviewer ZtLt2024-08-14

I will increase my score to 4, but I don't think my concerns regarding the limitations of the paper are addressed.

Area Chair 4KW52024-08-14

Discussions

Dear reviewers, Thanks very much for your great efforts in the review process. Please read the authors' response and confirm it. You are encouraged to discuss with all of us if you have any concerns. Thanks, AC

Reviewer Yksm2024-08-14

I thank the authors for addressing my questions. I will maintain my score. While I find the perspective of considering SAM as IGD interesting, I believe the paper should include more discussion on your technical contributions.

Authorsrebuttal2024-08-14

Dear Reviewer Yksm, Thank you very much. We will include more discussion of the technical contributions in the introduction, following your suggestions. Best regards, Authors

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC