Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach

Summary

Drawing inspiration from physics principles, the paper proposes the use of conservative Hamiltonian neural flows to construct GNNs that are robust against adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically evaluated on several benchmark datasets, considering a variety of adversarial attacks.

Strengths

This paper investigates the vulnerability of graph neural networks (GNNs) to adversarial perturbations that affect both node features and graph topology. The study focuses on GNNs derived from various neural flows and explores their connection to different stability notions.

Weaknesses

The followings are four weaknesses. 1. Paper's writing quality is poor due to excessive amount of equations, formulas, and its explanation in a view of physics. The authors should use the adversarial research language to bring out the main insight for why this work is important for adversarial research. 2. In addition, numerous words need to penetrate the core idea of equations rather than to revolve around technical terms. 3. The reviewer did not see the connection bridging the gap between Hamiltonian mechanics and energy conservation. 4. Is there any other simpler option that possess the notion of conservation other than Hamiltonian mechanics at all? In addition, why is it important to conserve energy in a view of topological structure? Overall, the reviewer recommends the authors to answer these questions using insightful language at least to the reviewer, not using mathematical or physical language. The reviewer does not want a theoretical proof, only want really contributional insight; Why is it important to consider energy or conservation and why are "energy and topology" bridged? Why is the hamiltonian important? Please answer using adversarial language --- The authors' rebuttal has fully explained the connection between the energy conservation in the realm of the Hamiltonian and the adversarial robustness of GNNs using the adversarial research language. Thank the authors for their thorough commentary on the reviewer's point.

Questions

Refer to Weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

1 poor

Contribution

3 good

Limitations

Refer to Weaknesses

Summary

This paper proposes a robust GNN model by leveraging the notion of Hamiltonian Energy Conservation. Specifically, authors first analyze the stabilities and limitations of several neural ODE-based GNNs, which motivate the proposed model HANG that is inspired by Hamiltonian classical mechanics. Experimental results indicate that HANG outperforms prior ODE-based GNNs against various adversarial attacks on several realistic graph datasets.

Strengths

- The paper is well written. All technical steps are easy to follow. - Authors propose a novel approach to improve GNN robustness. - Authors clearly motivate the proposed approach by analyzing the stability limitations of prior ODE-based GNNs. - The proposed model has been evaluated against various adversarial attacks.

Weaknesses

- The major issue of this work is that it only compares with a limited number of (and relatively weak) defense baselines. There are some recent methods such as [1, 2] showing better results than the baselines chosen in this paper. It would further improve the paper if authors could compare HANG with those methods. - Figure 1 only shows the norm of node features become closer as time goes, which does not support authors' claim that node features gradually become similar (lines 182-183). Authors may want to show the norm of relative difference of node features is approaching to zero instead. [1]: Geisler et al., “Robustness of Graph Neural Networks at Scale”, NeurIPS'21. \ [2]: Deng et al., “GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks”, LoG'22. I'm willing to raise the score if my major concerns are addressed.

Questions

- Is it possible to further improve GNN robustness by combining HANG with some prior defense methods (e.g. graph preprocessing-based methods), as they improve robustness in orthogonal ways?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the Weaknesses section above for limitations.

Summary

This paper explores the robustness of Graph Neural Networks (GNNs) against adversarial attacks. Drawing inspiration from principles in physics, the authors propose a novel model called Hamiltonian Neural Flows for constructing GNN models. The effectiveness of the proposed method is evaluated on various benchmark datasets.

Strengths

The problem addressed in this paper is both significant and intriguing, and the proposed method is well-supported by principles from physics.

Weaknesses

I am a non-expert in physics-based methods, my evaluation is based on my understanding of GNN methods and educated guesses. I suggest AC to ignore my evaluation if there are expert reviewers in the field. Here are a couple of suggestions to improve the paper: In my opinion, the comparison with existing baselines seems inadequate. The TDGIA paper was published in 2021, while MetaGIA was published in early 2022 or late 2021. It would be beneficial if the authors could include more recent graph attack papers and compare their proposed method against them. Additionally, it would be interesting to see a comparison between the proposed method and the Graph Isomorphism Network (GIN).

Questions

see cons above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

Since neural ordinary differential equation networks can show inherent robustness, in this work the authors try to perform an extensive study on different graph neural flows along with their stability on different stability notions like BIBO stability, Lyapunov stability, Structural stability and Conservative stability. The authors find that the graph neural flows using hamiltonian energy functions can achieve improved empirical adversarial robustness on using black box attacks like PGD and TDGIA. The authors show that designing graph neural flows which can ensure conservation stability along with Lyapunov stability can help in achieving improved adversarial robustness. As common in literature the authors generate a white box attack from a surrogate model and then transfer it to the black box model to testify the robustness of the black box model. Evaluation is done on node injection attacks as well as graph manipulation attacks, and significantly improved robustness is seen in both cases.

Strengths

* I think using graph neural flows to understand if we can achieve inherent robustness without the need for adversarial training is interesting. * The way the authors have related different stability criteria with robustness is quite interesting. * The results show significant improvements.

Weaknesses

* There have been many instances where defences claiming to be robust have been later evaded using adaptive attacks [1]. Adversarial training has been the most successful defence strategy and the current successful defences use adversarial training to achieve robustness. Therefore, it is necessary that the robustness of the proposed method is evaluated properly. I think that the surrogate model-based black box attacks are not strong enough to get the worst-case robustness. Therefore, it is important that the evaluation on strong white-box attacks like PGD, PGD with max-margin loss / Carlini and Wagner attack [2] is used to testify robustness, and this evaluation should be carried out in a white-box setting. Further, it is important that the authors share a robust accuracy (both white box and black box) vs constraint threat model plot for the proposed approach, GAT and GAT trained using PGD-AT [3]. * Comparison is done only with models trained using standard training. It is important to include adversarially trained models as baselines. For instance, the authors should include a PGD-AT trained model as a baseline in all the tables. * In the case of GNNs it is important to ensure that the attack graph remains imperceptible and the attacked nodes in the case of node injection attacks cannot be pruned off. Therefore other than robust accuracy imperceptibility of the graph for a given threat model is also a very important metric to be considered. Similar to [], I request the authors to also include a comparison on the imperceptibility of the attack on the proposed Graph neural flow network and the baselines like GAT. It is important to ensure that the imperceptiblity of the proposed defence is either lower or equal to the baselines against the PGD and TDGIA attacks. * The authors should also look at these works [4,5] and try to compare their methods with them . [1] Athalye, Anish et al. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.” International Conference on Machine Learning (2018). [2] Carlini, Nicholas, and David Wagner. "Towards evaluating the robustness of neural networks." [3] Madry, Aleksander et al. “Towards Deep Learning Models Resistant to Adversarial Attacks.” [4] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.” [5] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.”

Questions

t would be great if the authors can address the raised concerns in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes the authors seem to have addressed the limitations. I would suggest the authors to kindly look at the weakness section for suggestions and comments.

Follow-up

Thanks for the detailed response and additional experiments, which clearly demonstrate the advantages of the proposed model. I would encourage authors to incorporate the clarification on the primary focus of this paper, as well as new empirical results of HANG+defense methods, in the revision, which would definitely improve the paper. Since authors have addressed my major concerns, I raise my score to 7.

Reply to Authors

I am not sure if the authors have evaluated their method properly on the adaptive attacks. Other answers given by the authors seem to satisfy my concerns and therefore, I would like to raise my score to 6.