Response 1/2
We appreciate a lot for your careful review, and would like to provide responses to your mentioned questions and weaknesses one by one.
###**[W1 & Q1 - Is GraphSAM only a good approximation for graph transformers? How is GraphSAM working, e.g., in the image domain? — Yes, GraphSAM only works good at molecular graph domain.]**
It is experimentally found that GraphSAM is a good approximation for graph transformers rather than in CV domain. Although our proposed method is designed based on the empirical observations, actually, Observations 1 and 2 are presented only in the graph transformers, while they do not appear in those of CV domain [1]. In CV, the model's perturbation gradient variations are often consistent with updating gradients, which is not in line with the phenomenons on the molecular graphs.
To further address the concern and provide holistic discussion, we add the following experiments to compare the performance of lookSAM [1], AE-SAM and GraphSAM in the CV and molecule datasets, respectively. LookSAM and GraphSAM are efficient SAMs designed based on the gradient varification patterns of models in their respective domains, and AE-SAM is an adaptively-updating efficient method accoding to the comparison evaluation between gradient norm and a pre-defined threshold.
|ResNet-18 (CV) | CIFAR-10 | CIFAR-100|
| :----: | :---: | :---: |
| +SGD| 95.41 | 78.21|
| +SAM| **96.53** | 80.18|
| +LookSAM | 96.28 | 79.91|
| +AE-SAM | 96.49 | **80.31**|
| +GraphSAM | 95.86 | 78.69 |
|CoMPT (Graph) | Tox21 | Sider| ClinTox|
| :----: | :---: | :---: | :---: |
| +Adam| 82.81 | 62.17| 91.41|
| +SAM| 83.96 | 64.33| 92.73|
| +LookSAM | 82.55| 62.54| 91.64|
| +AE-SAM | 83.33| 63.16| 91.92|
| +GraphSAM | **84.11** | **64.58** |**93.78** |
It is observed LookSAM has better results on the image datasets instead on the molecular graphs investigated in this work. On the contrary, our GraphSAM works on the molecules but leads to the worst performances on the image benchmarks. That is because model gradient variations are diverse accross the different domains. It is challenging to transfer the efficient SAMs designed based on the specific gradient patterns. AE-SAM achieves an acceptable performance on the molecular graphs since the perturbation gradient norms of graph transformers are monitored to inform the necessarity of gradient re-computation. But it is not as good as GraphSAM where we accurately fit the perturbation gradients at each step. In summary, GraphSAM is optimized particularly for the graph transformers based on the empirical observations of gradient variations.
###**[W2 - The conclustion of Theorem 1 is not clear. — Will add to the following new explanations.]**
Thank you for your constructive feedback regarding the theoretical aspects of our paper. We hope the following revisement can mitigate your concerns.
The revised sentence is: "Recalling the min-max optimization problem of SAM in Eq. (1), if we replace the inner maximum objective from $L_{G}(θ + ε^{S})$ to $L_{G}(θ + ε^{G})$, the graph transformer is motivated to smooth a worse neighborhood loss in the loss landscape. In other words, the proposed GraphSAM aims to minimize a rougher neighborhood loss, whose value is theoretically larger than that of SAM, and obtain a smoother landscape associated with the desired generalization."
###**[W3 - Proof of Theorem 1 relies on empirical observations and strong assumptions. — Agreed, we have now changed it.]**
Thanks for your suggestions, and we have uniformly changed 'Theorem' to 'Conjecture'. Conjecture is a statement or conclusion based on insufficient evidence, rather than through rigorous proof. We rewrite Theorem 1 to include the experimental assumption that supports our conclusion.
The new Conjecture 1 is:
Let $ϵ̂_{S}$ and $ϵ̂_{G}$
denote the perturbation weights of SAM and GraphSAM, respectively,
where we ignore the subscript of t for the simple representation.
Suppose that ω/|ω| >> ϵ ,as empirically discussed in Observation 1, and
$|ϵ̂_{S}|$ < $|ϵ̂_{G}|$
for ρ> 0,
designating $ϵ̂_{S}$ as the ground-truth. We have:
L_{G}(θ+$ϵ̂_{S}$)
≤
L_{G}(θ+$ϵ̂_{G}$)
**References:**
[1] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, Yang You. Towards efficient and scalable sharpness-aware minimization. CVPR 2022.