Summary
**I have read the author's rebuttal, see reply below** This work proposes changing SAM from perturbing the weights by the gradient of the current minibatch, with instead a moving average of the gradients across training iterations. They provide upper bounds that indicate that the moving average may better approximates the inner-maximization with respect to the loss over the entire dataset, which is more difficult for naive SAM due to the minibatch noise. Empirically, SAM with moving average gradients (which they call VaSSO) performs better on CIFAR10, CIFAR100 by around $0.1-0.5$%, and the improvements are more apparent in the presence of heavy label noise with gains up to +10% improvement with 75% label noise. Empirical experiments show that VaSSO minimizes the maximum eigenvalue of the Hessian better than SAM.
Strengths
**Significance**: The proposed algorithm VaSSO is a very simple change that achieves flatter minima and achieves better generalization than SAM. Although it may not necessarily be the case that two effects are tied in a causal manner, its effectiveness at achieving both beyond SAM may suggest that it may be a useful optimizer in practice and for further study in future works.
**Quality**: The paper is clearly written and organized.
Weaknesses
**1. Lack of important literature review**: The authors motivate VaSSO by identifying problems with SAM in terms of its ability to optimize the original objective _due to the minibatch noise_. The moving average is hypothesized to be better at correctly approximating the full-batch gradient. The authors do not address the fact that works that have analyzed SAM previously have unanimously observed that SAM actually **only observes improvements in generalization if it is paired with minibatch noise**. In particular, Andruschenko et al (https://arxiv.org/abs/2206.06232) showed that n-SAM which utilizes the full-batch gradient directly for the perturbation step actually observed no generalization gain. This is supported by experiments on m-sharpness in the original SAM paper (https://arxiv.org/abs/2010.01412), and they also showed that better approximating the sharpest direction by also taking the second order approximation of the loss instead of the first order approximation lead to worse performance. Wen et al (https://arxiv.org/abs/2211.05729) showed that the minibatch noise is important to minimize the trace of the Hessian instead of the max eigenvalue (though the work does not make any claim about which measure of sharpness is better correlated with generalization).
To summarize, the authors pose the problem as that SAM cannot achieve the full generalization benefits because it does not minimize its intended objective optimally, but previous literature indicate that this suboptimality is actually what allows SAM to achieve better generalization. The authors do not mention the conclusions made in previous works, but it seems important to address this conflict.
**2. Marginal improvements, and inconsistent numbers in comparison to Foret et al.**: Without label noise VaSSO's improvements range between 0.2-0.5% for CIFAR10 and CIFAR100, and it's not clear whether further hyperparameter tuning could close this gap. In particular, the authors train WideResNet-28-10 on CIFAR10 with cutout data augmentation which was also conducted by Foret et al, where they report an error of 2.3% while the authors report 2.7% and for VaSSO 2.5%. This slight improvement may potentially come from the authors only optimizing the rho hyperparameter, and not m-sharpness (the other hyperparameter mentioned in Foret et al. where the minibatch is sharded first).
For label noise, there is a much more significant boost in test accuracy, but it is unclear whether the authors are reporting the peak early-stopping accuracy or the final accuracy. The former is more important for label noise, and the difference between VaSSO and SAM may be coming from reporting the latter. Also, Foret et al reported better numbers for SAM on CIFAR10 with similar amounts of label noise although they were using ResNet34 instead of ResNet18.
**3. The upper bounds are too loose, and comparing upper bounds does not lead to any meaningful comparison.** The upper bound derived in Theorem 2 requires very approximate intermediate steps (very loose upper bound in Line 181), and is not a function of $t$ but $T$. The bound implies that if you train for longer, the bound for the MSE between the moving average and the full batch gradient _at every intermediate step_ improves which doesn't seem right.
More importantly, any meaningful comparison between the MSE for SAM and MSE for VaSSO should compare the upper bound of VaSSO to the _lower bound_ for SAM. Comparing two upper bounds isn't very meaningful.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.