Further response to Reviewer c6Fm
We sincerely thank the reviewer for the response. We would like to address the remaining concerns:
> 1. Motivation
We kindly ask in what way our response for Q5 in the rebuttal did not clarify the motivation.
In rebuttal, we explained that the particular choice of $y_{t+1} = x_t - \eta_t \nabla f(y_t)$ is a direct consequence of the analysis, which ensures that the discrepancy between $\nabla f(x_{t})$ and $\nabla f(y_{t})$ can be controlled. Empirically, the small difference between $\nabla f(x_t)$ and $\nabla f(y_t)$, along with similar Hessian spectra and test accuracy between SAM and SAMPa-0, confirms that $\nabla f(y_t)$ is an excellent approximation of $\nabla f(x_t)$.
Additionally, Section 2.2 and Section 3 provide detailed explanations of our intuition and the development of SAMPa. Key points include:
- *Perturbation direction matters:* RandSAM, which uses random perturbation, performs worse than SAM, highlighting the importance of perturbation direction.
- *Using past gradients is ineffective:* OptSAM, using past gradients for perturbation, underperforms than SAM and it even fails to converge in a toy example due to its reliance on a poor gradient estimate from an ascent step.
- *Introduce the auxiliary sequence:* We finally propose an auxiliary sequence $y_{t+1} = x_t - \eta_t \nabla f(y_t)$ as a direct result of our theoretical analysis, which aligns with the descent direction from $x_t$.
> 2. Theory limitation
We do not understand why convex analysis should be less suitable for SAM than e.g. the mentioned AdaGrad or Adam. SAM has mostly seen success in overparameterized settings (where the training data is eventually perfectly fitted), where it has been exploited in several works where convexification can occur. Additionally, the solution set for the convex problem is not necessarily a singleton, so biasing the obtained solution still seems reasonable.
We hope our clarification addresses all your concerns, and we are happy to discuss them if you have any further questions.
We would like to emphasize that SAMPa significantly advances SAM by breaking the sequential nature of its two gradient computations, enabling parallel execution. SAMPa not only maintains convergence guarantees in the convex setting but also, as shown in our empirical results, achieves the fastest computational time among five efficient SAM variants (including SSAM introduced in the rebuttal) while consistently improving generalization across multiple tasks. We kindly ask the reviewer to consider these contributions.
Thank you again for your engagement!