Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima

Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + ρ\frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size $ρ$ and/or no gradient normalization in $y_t$, which is detached from practice. To address this gap, we study deterministic/stochastic versions of SAM with practical configurations (i.e., constant $ρ$ and gradient normalization in $y_t$) and explore their convergence properties on smooth functions with (non)convexity assumptions. Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points. For smooth strongly convex functions, we show that while deterministic SAM enjoys tight global convergence rates of $\tilde Θ(\frac{1}{T^2})$, the convergence bound of stochastic SAM suffers an inevitable additive term $O(ρ^2)$, indicating convergence only up to neighborhoods of optima. In fact, such $O(ρ^2)$ factors arise for stochastic SAM in all the settings we consider, and also for deterministic SAM in nonconvex cases; importantly, we prove by examples that such terms are unavoidable. Our results highlight vastly different characteristics of SAM with vs. without decaying perturbation size or gradient normalization, and suggest that the intuitions gained from one version may not apply to the other.

Paper

Similar papers

Peer review

Reviewer y8UR6/10 · confidence 4/52023-06-23

Summary

This paper analyzes the convergence properties of Sharpness-Aware Minimization (SAM) with **constant** perturbation size $\rho$ and gradient normalization applied to the updates; this has not been done in prior works. Both deterministic and stochastic settings are considered. The authors show the following: 1. For the strongly convex + smooth case in the deterministic setting, SAM converges to the global minimum. A matching lower bound for the rate is also presented although I'm not sure if it's correct; please see Weakness #2. For the general convex + smooth case in the deterministic setting, the authors are only able to show convergence to a stationary point (in terms of gradient norm). 2. More importantly, the authors show that in all other settings considered in the paper, SAM cannot converge to the global/local minimum. Specifically, this includes the smooth non-convex and non-smooth Lipschitz convex cases in the deterministic setting and smooth + (strongly) convex and smooth non-convex cases in the stochastic setting. Some lower bounds are also presented to complement the upper bounds. In summary, compared to the case of decaying perturbation size $\rho$ analyzed in prior theoretical works on SAM, this work shows that using a constant $\rho$ can inhibit convergence.

Strengths

**1.** Unlike prior theoretical works on SAM, this paper considers the use of constant perturbation size and gradient normalization in the SAM update which is more aligned with practice. **2.** The theory is comprehensive in the sense that the common function classes are covered and there is analysis for both the deterministic and stochastic settings. **3.** In some cases, matching lower bounds are also presented to support the upper bounds. Some of the constructions and techniques for the lower bounds seemed novel to me (for e.g., Theorem 3.6) but I'm not an expert on lower bounds. **4.** The paper is also written more or less clearly.

Weaknesses

**1.** In the abstract, the authors write: "*Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points*". But I don't find the non-convergence of SAM **with constant $\rho$** (and constant step-size $\eta$) very surprising; in fact, the asymptotic convergence in Theorems 3.1 and 3.3 surprises me. I say this because of the following reason. Suppose deterministic SAM converges to some point $\tilde{x}$. Then, we must have $\nabla f\Big(\tilde{x} + \rho \frac{\nabla f(\tilde{x})}{||\nabla f(\tilde{x})||}\Big) = 0$. This means that we must have $\tilde{x} + \rho \frac{\nabla f(\tilde{x})}{||\nabla f(\tilde{x})||} = x^{\ast}$, where $x^{\ast}$ is some stationary point of $f$. So, $|| \tilde{x} - x^{\ast}|| = \rho$, i.e., if SAM with constant $\rho$ converges to some point, then that point must be $\rho$ distance away from a stationary point of $f$. Notice that if $\rho$ is a decreasing function of $t$, this (apparent) issue of non-convergence won't arise. **2.** There seems to be an issue in Theorem 3.2 which states that $\frac{\beta}{\mu} \geq 2$. But in the proof of Theorem 3.2, one-dimensional quadratics are used to obtain lower bounds; for these one-dimensional quadratics the **tightest possible** smoothness constant = **tightest possible** strong convexity parameter. So actually, $\beta = \mu$ in all 3 cases. Now in Case 1, $x_0 = 2\rho$. But then, we get $x_t \geq \frac{x_0}{2} = \rho$. Unfortunately, this yields a vacuous lower bound of 0. Can the authors comment on this? It seems that if the current proof strategy is to be used, we would need at least 2-d quadratics where $\beta > \mu$. **3.** In Theorem 3.4, the dependence of the convergence bound w.r.t. $T$ seems loose and I don’t think the Lipschitz assumption is required. In the second equation of the proof, there is a $-\frac{\eta}{2}||\nabla f(y_t)||^2$ term and a $\frac{\eta^2 \beta}{2}||\nabla f(y_t)||^2$ term; these two terms can be combined together and as long as $\eta \beta < 1$, the coefficient of $||\nabla f(y_t)||^2$ is negative. So I think we can get $O(1/T)$ convergence to a $O(\beta^2 \rho^2)$ stationary point with a fixed step-size independent of $T$. Also, if we follow this approach, we don’t even Lipschitzness. **4.** It is unfortunate that the lower bounds for the stochastic case needed to be corrected in the supplementary material. **5.** (Minor weakness but a limitation nevertheless) SAM was proposed to improve generalization. But this paper has no results illustrating how SAM improves generalization such as by converging to points that are *approximately* flat minima (or because it inhibits convergence preventing over-fitting). A missed reference (just for the author's information): https://openreview.net/pdf?id=IcDTYTI0Nx shows the convergence of SAM with non-constant perturbation sizes.

Questions

Please address Weaknesses 1, 2 and 3. My current rating is mainly due to Weakness 1; if the authors can point out some fallacy in my understanding or convince me that the *non-convergence* of SAM isn't all that obvious, I can increase my score.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Some minor limitations have been discussed such as the loose dependence on $\beta$ and $\mu$ in Theorem 3.1. But important limitations such as Weakness #5 have not been discussed. No foreseeable negative societal impacts.

Reviewer T74n7/10 · confidence 3/52023-06-25

Summary

SAM is a very practical algorithm for improving generalization in deep learning, however, even the convergence properties of SAM are not well understood. The paper studies the convergence/non-convergence of SAM under various standard setups in optimization, including smooth/nonsmooth, convex/strongly convex/nonconvex, deterministic/stochastic optimization problems. The results of this paper provide a detailed and complete description of the convergence of SAM for these problems.

Strengths

1. Compared with existing works, the paper analyzes SAM with fixed $\rho$ and normalization steps, which is the algorithm that people use in practice. Under these practical configurations, this paper provides different characteristics of SAM. 2. The upper bounds give sufficient conditions for the convergence of SAM and the convergence rate of the algorithm under these conditions. Additionally, this paper also provides lower bounds to show the tightness of their upper bounds in the dependency of certain important problem parameters such as $\rho$ and $T$. 3. I found the lower bounds very interesting and they provide many insights about SAM. In contrast to gradient descent (GD), the result in this paper shows that deterministic SAM (full-batch SAM) fails to converge to a stationary point even for smooth nonconvex objectives. In contrast to stochastic gradient descent (SGD), stochastic SAM (m-SAM) even fails to converge to a global minimum for smooth strongly convex objectives. These results justify the limited capability of SAM in optimization and suggest the differences between SAM and other common optimizers such as SGD/Adam/RMSProp.

Weaknesses

See the questions part.

Questions

1. It seems that we only have the non-convergence result of stochastic SAM for smooth and strongly convex functions for m-SAM. Can we prove similar lower bounds for n-SAM? If not, I hope the authors can explain the underlying intuition. By the way, it is proper to say "stochastic SAM for every function classes we consider, fail to converge properly" in Line 359? 2. I can not fully understand the purpose of Sec. 3.4 as well as Thm. 3.6. It seems that for merely convex objectives, gradient descent also fails to converge to a global minima? See the second inequality in Thm. 2.1.7 in Nesterov's celebrated book Lectures on Convex Optimization", Second Edition.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

n/a

Reviewer NH1R5/10 · confidence 3/52023-07-02

Summary

This paper studies the convergence properties of sharpness-aware minimization in a specific setting with the use of gradient normalization and arbitrary constant perturbation. The paper established convergence rates for both deterministic and stochastic SAM with various assumptions on the convexity of the objective function. The authors argue the term $\mathcal{O}(\rho^2)$ appearing multiple times in their bounds is in fact unavoidable by establishing corresponding lower bounds.

Strengths

- The paper is well-written and easy to follow. - The analysis covers different levels of convexity. - The author proves lower bounds to justify their proposed convergence rate.

Weaknesses

- The proof of the upper bounds seems to have followed that of [1]. - Several cases are left out, including the lower bound for the smooth and convex case of deterministic SAM, and the lower bound for the smooth and strongly-convex case of stochastic SAM with small variance. - Generalization is not considered.

Questions

- General questions: - The paper considers SAM with constant stepsizes and gradient normalization and this setting seems to be a common choice in practice. However, I do think more justification for this setting is needed, especially when [1] derived faster convergence rates for SAM without gradient normalization. - It seems no methodological conclusions for the implementation of SAM are provided in the paper. Based on the results in this paper, (how) should we decrease the perturbation $\rho$ in the algorithm so that the additive term $\mathcal{O}(\rho^2)$ would be resolved and thus the convergence be faster? - Small issues: - There is only one lower bound result in Table 1, which looks weird to me. - In line 830, does "Assumption 4.2" refer to Theorem 4.2? [1] M. Andriushchenko and N. Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pages 639–668. PMLR, 2022.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Reviewer AJAE7/10 · confidence 3/52023-07-10

Summary

This paper examines convergence properties of sharpness-aware minimization, which is an empirically-popular algorithm for training deep neural networks, yet whose theoretical properties are poorly understood. This paper makes several contributions to the analysis of sharpness-aware minimization, which are novel in this growing literature. For smooth and convex functions, this paper proves that the best iterate of SAM converges to the global minimizer. This is expected as, for such functions, there exists a global minimizer; thus, even after perturbations, the convergent point should still be the global minimizer. For smooth and nonconvex functions, this paper proves that there is an additional bias term in the convergent point compared with a local minimizer. Furthermore, a lower bound example is constructed, showing that this bias term is necessary.

Strengths

- The paper is nicely written and provides enough background knowledge for readers to understand the topic. For each theorem, the authors accompany their results with a sketch of the high-level proof arguments. - The lower-bound examples are especially interesting in terms of complementing the convergence rates.

Weaknesses

- The results in the main text are a bit dense. It would be easier to read if the paper is formatted in a more friendly manner. - There are no experiments/simulations to complement the theoretical results. I consider this a limitation since SAM is primarily motivated as an empirically-successful algorithm, so a connection between theoretical results to their practical implications would be important for this audience.

Questions

- I wonder if the authors could discuss the practical implications of their findings. For example, how does the extra bias term affect the empirical performance of SAM; Does it inject some kind of regularizer that is actually helpful for the empirical performance, or is it possible to design a better algorithm that eliminates this bias term altogether? - There is a concurrent paper that studies a very related question: Dai, Y., Ahn, K., & Sra, S. (2023). The Crucial Role of Normalization in Sharpness-Aware Minimization. arXiv preprint arXiv:2305.15287. It would be helpful if the authors could discuss the relation to this paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors discuss the limitations of their results along with presentations in the main text. The potential societal impact of their work should be minimal, given the technical nature of the paper.

Reviewer T74n2023-08-11

Repsonse to the Rebuttal

The author's rebuttal addressed my concern well. I think it is a very interesting paper, and I decide to raise the score to 7.

Authorsrebuttal2023-08-12

Additional response to Reviewer T74n

We express our gratitude to the reviewer for the valuable discussion and positive evaluation. We are glad to hear that our discussion has cleared your concerns. Please let us know if there are any further inquiries. Best regards, Authors

Reviewer y8UR2023-08-13

Reply to Rebuttal

I thank the authors for the detailed rebuttal! **Weakness 1**: Thank you for the explanation. Gradient normalization becoming 0 at stationary points is somewhat disconcerting to me. I see that under this seemingly unrealistic definition of normalization, non-convergence is not all that obvious. But I feel this definition is misaligned with reality and hard to reconcile. And perhaps that is why it would be meaningful to incorporate the small corrective $\epsilon$ term in the denominator of normalization. I will raise my score to 6 but I strongly encourage the authors to discuss this limitation and include all the discussion here in the next version of the paper. **Weakness 2**: I agree that the bound can be corrected **w.r.t. $\rho$** by properly up-scaling $x_0$ but the factors of $\beta$ and $\mu$ in the bound don't seem right to me because the functions are 1D quadratics with $\beta = \mu$; I suggest removing the $\frac{\beta}{\mu} \geq 2$ condition in the theorem statement as well as the $\beta$ and $\mu$ terms in the bound.

Authorsrebuttal2023-08-15

Additional response to Reviewer y8UR

We appreciate the reviewer for providing thorough insights, and we are glad to know that the reviewer decided to offer a positive evaluation; we find it highly encouraging. As for Weakness 1, we agree to the reviewer that setting $\frac{\nabla f(x)}{\lVert \nabla f(x) \rVert} = 0$ when $\nabla f(x) = 0$ could be deemed a non-standard way of normalization. However, we believe it is reasonable to adopt this approach when it comes to analyzing practical version of SAM. This is because the practical implementations of SAM in fact use $\frac{\nabla f(x)}{\lVert \nabla f(x) \rVert + \epsilon}$, where a small $\epsilon > 0$ is introduced for numerical stability (e.g., the official code of SAM in Tensorflow). We will thoroughly address this discrepancy in our next revision to ensure it is not overlooked. Regarding the issue raised in Weakness 2, we would like to re-emphasize that the $\beta$ and $\mu$ serve as constants defining the **function class** $\mathcal F$, rather than **individual functions** chosen from the class. It is important to note that a 1-d quadratic function $f(x) = \frac{1}{2} a x^2 + b x + c$ is by definition $\beta$-smooth and $\mu$-strongly convex for **any** $\beta$ and $\mu$ satisfying $\mu \leq a \leq \beta$. For instance, the function $f(x) = \frac{1}{2}\cdot\frac{3}{4}x^2$ is $\frac{1}{2}$-strongly convex and $1$-smooth, although these constants are obviously not the tightest. In the proof of Theorem 3.2, our goal is to find the worst-case examples out of the function class $\mathcal F$. This class contains all $\beta$-smooth and $\mu$-strongly convex functions, where $\frac{\beta}{\mu} \geq 2$. Since the **worst-case** functions chosen in our proof are 1-d quadratic functions, with coefficients of quadratic term being $a = \mu$, $\frac{\beta}{2}$, and $\beta$ for Cases 1, 2, and 3, respectively, it becomes evident that $\mu \leq a \leq \beta$ holds for all three cases and they are indeed valid members of $\mathcal F$. Having said that, we acknowledge that it is quite confusing, and it is ideal not to have the assumption $\frac{\beta}{\mu} \geq 2$. We will carefully reconsider it and make necessary adjustments to improve our result and minimize any possible confusion. Again, we appreciate your valuable feedback. We will make sure to incorporate all the discussions into the next version of the paper.

Reviewer NH1R2023-08-15

Thanks for the response. I am raising my score to (5), subject to the authors adding the above comments in their paper.

Authorsrebuttal2023-08-15

Additional response to Reviewer NH1R

We sincerely appreciate the reviewer for their thorough discussions and for offering a positive evaluation. We will ensure that all the above discussions are incorporated in the next version of the paper. Best regards, Authors

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC