Summary
This paper proposes a bracket-based dynamical system framework to design structure-preserving graph neural networks (GNNs). Specifically, the authors leverage four formalisms: Hamiltonian, Gradient, Double-Bracket, and Metriplectic, that model physical systems with different completeness and dissipation characteristics, and implement them via discretizations from exterior calculus. The authors parameterize GNNs with a chosen dynamic by specifying the bracket matrices and learning the energy/entropy functions via graph attention mechanisms. Experiments on real-world graph inference tasks show that structure-preserving GNNs can perform better than those that are not strictly structure-preserving, and the best dynamics depend on the downstream task.
Strengths
Strengths:
1. It is novel to use bracket-based parameterizations to design physics-inspired GNNs.
2. It is fruitful to analyze existing GNNs (e.g., GAT, GRAND) under the bracket-based dynamical framework.
Weaknesses
Weaknesses:
1. Since the paper is intended for general audience in graph machine learning, it can be written in a clearer manner with more motivating examples and targeting applications. For starters, include a notation section.
2. One of the key motivations of the paper is to improve performance of deep GNNs, yet all the experiments in the main manuscripts do not investigate explicitly the role of depth - are the GNNs used in these experiments actually deep? The more related experiments are described and discussed in Appendix B.3.2 as ablation study, but results (Table 8) show that for node classification task, only Gradient-based GNNs perform well when the number of layers increases, whereas other dynamics exhibit (significant) performance drop. If the paper intends to mitigate over-smoothing, the authors should provide more compelling experiments. On the other hand, if the paper simply intends to describe a physics-inspired framework to parameterize GNNs, then the authors should re-write the discussion of deep GNNs.
Questions
1. What's the implications for practitioners? What applications would we expect imposing certain bracket structure outperforms the unconstrained ones? What structural requirement one should choose for different applications (e.g., add a column in Table 1 giving example applications)? The current set of experiments reads a bit confusing to me. For example, experiment 2 considers the node classification task, which requires some sense of clustering / smoothing, but not over-smoothing. So the findings in Table 4 showing that Double Bracket performs the best aligns with our expectations? But in Table 8, the Double Bracket formalism cannot be computed for deeper architectures - Is it merely a computational constraint? More explicit remarks and discussions will be great!
2. The role of (learnable inner product) $A$: the authors connect this with graph attention mechanism; but a priori one can also choose other forms of $A$? Have the authors compare some simple baselines, say fixed $A$ or even just set $A$ as identity? This could also help address the memory issue in experiments (Sec 5.3) where the authors cannot compute $A$ (and thus $E, S$) due to the high-dimensionality of the node features?
3. The authors discussed higher-order attention mechanism based on higher-order cliques (Appendix 4). Will this be useful in practice or is this a pure theoretical discussion? If the former, can the authors provide experimental demonstration? If the latter, the authors might consider removing this from the primary contributions since this does not seem like the focus of the paper?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
This work does not have negative societal impact. The limitations are discussed in Section 6.