Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?

Summary

The present work pertains to analysing and improving Graph Neural Networks operating on geometric graphs that possess E(3) (Euclidean) symmetries, i.e. the tasks of interest are invariant/equivariant to rotations, translations and reflections. The paper challenges the assumption that it is sufficient to use E(3)-equivariant features of 1st degree (the term “degree” is related to the dimensionality of the group representation to which the features are equivariant - 1st-degree features are 3-dimensional and are equivariant to 3D rotations/reflections), as done in EGNN, Satorras et al., ICML’21, which works quite well in practice and is computationally efficient. The authors construct a series of counterexample symmetric geometric graphs (i.e. graphs which, when rotated by certain angles, remain intact up to permutations of their vertices) which are shown to be challenging for geometric GNNs. It is shown that, for certain degrees $l$, a GNN can only produce output features equal to zero, therefore showing that the degree is crucial for expressivity and that $l=1$ is insufficient for some cases. To address this limitation, they combine ideas from EGNN and TFN, Thomas et al., arxiv’18: They create a GNN, similar to EGNN, enriched with auxiliary features of higher degree, using spherical harmonics (similar to TFN). To keep the computational complexity low, they produce the messages used for the updates of node features/positions/auxiliary features via scalarisation using inner products, simplifying the messages used in TFN. The expressivity of the method is theoretically analysed, showing the ability to recover angles, and empirically tested, showing improved performance in several synthetic and real-world benchmarks.

Strengths

**Significance**. The studied problem is important for various physical sciences using data-driven methods and finding better trade-offs between performance and computational complexity is key to their progress. The present paper improves this trade-off both theoretically and empirically. **Novelty**. - *Theoretical results*. The underlying theory illustrating the expressivity barriers of low-degree representations (theorems 3.4-3.6) is simple and concise, yet insightful and of independent technical interest (can be used for other types of symmetric objects to find out properties of neural representations and illustrate limitations of modelling choices). Additionally, theorem 4.1. backs the proposed method with theoretical arguments, hinting that is probably a good trade-off between expressivity and computational complexity (although the former is not completely characterised). - *Method*. As far as I know, the proposed approach to incorporate higher-degree representations into message-passing is original. Additionally, it is easy to understand and implement, which may facilitate reproducibility. **Presentation**. Apart from a few exceptions (see weaknesses), the paper is well-explained, with carefully chosen notation and progressive illustration of the different results (from the theoretical motivation to the model improvement and empirical evaluation). **Empirical results & Execution**. The experimental outcomes are in agreement with the theory on synthetic data, while the method is shown to generalise better than its competitors in real-world data, with additional computational efficiency advantages. Overall, this work is well-executed: starting from a characterisation of the limitations of existing works, then proposing a remedy (that gets the best of both worlds from two different families of methods) and finally testing it both on the counterexamples, as well as in real-world scenarios, validating the theory.

Weaknesses

**Clarity: limits audience and creates some issues with contextualization to related work**. Although not a major weakness, there are some issues with the clarity of the presentation, which I think should be addressed to improve the exposition of the paper to a wider audience. - In the introduction and the related work section, the authors mention several concepts without providing explanations which might make the text hard to follow, especially for readers not experts in geometric GNNs and Euclidean symmetries. - For instance: *x-degree steerable representations* - this is a notion at the heart of the paper, so it should probably be intuitively explained early on, e.g. by summarising the formal definitions at the end of section 3.1. - *Clebsch-Gordan tensor product* - this concept is not explained in the paper. However, I do not think that the authors should assume familiarity with it. The reader needs to obtain a clear understanding due to the following: the main methodological innovation of the present work is to replace this operation with inner products to improve computational efficiency. I think that the authors should explain in a separate subsection the differences between these two operators and discuss what they sacrifice by not incorporating CG products in their architecture. - Examples 3.2. and 3.3. need further elaboration as they are the main motivation behind the proposed methodology. E.g. why do even folds are not symmetric w.r.t. the cyclic group? Perhaps explain what the dihedral group is. Why are regular polygons symmetric w.r.t. the dihedral group?

Questions

- Section 3.1/Section 4: It is unclear to me why the authors introduced the modulation of the spherical harmonics (re-scaling). How does this affect expressivity? - Section 3.1: Is the representation of O(3) – Eq. (2) – conventional or introduced here for the first time? In the first case, could the authors provide a reference? In the second case, it would be more appropriate to precisely show that this is indeed a representation for completeness. - As far as I understand, Eq. (3) refers to graph-level features. Perhaps this should be explicitly mentioned to improve readability. - What information is lost from the scalarization trick of Eq. (6) compared to CG tensor products? To what extent does this limit expressivity? - How do the authors explain that in certain cases in Table 2, HEGNN does not achieve 100% performance (although it should according to the theory)? Additionally, why is the std so high (15% and 30%)? - In table 3, it seems that HEGNN with $l\leq 1$ always outperforms EGNN, which is also 1st-degree. How do the authors explain this? - It is unclear to me how equivariance to reflections is handled by this method. How does the discussion at the end of section 3.1 affects the operators used? - Table 5 is quite helpful, perhaps consider moving it to the main text. **Minor**: I have the impression that there were a few typos throughout the text. I suggest performing a proof-reading (examples: L78 typo: university --> universality, L266 type: donated --> denoted).

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

I recommend that the authors devote a separate section to discuss the limitations of their work. Currently, I could locate only a limited discussion in the appendix with regards to not testing in large-scale systems. I do not foresee any negative societal impact.

Summary

This paper challenges the prevailing notion that high-degree steerable representations are unnecessary in equivariant Graph Neural Networks (GNNs). The authors provide theoretical analysis showing that equivariant GNNs constrained to 1st-degree representations degenerate to zero functions when applied to symmetric structures like k-fold rotations and regular polyhedra. To address this limitation, they propose HEGNN, a high-degree extension of the EGNN model that incorporates higher-degree steerable vectors while maintaining efficiency through a scalarization technique. The authors evaluate HEGNN on symmetric toy datasets, N-body systems, and MD17 molecular dynamics datasets, demonstrating improved performance over existing models.

Strengths

1. The paper presents a clear and well-motivated research question, challenging an existing assumption in the field of equivariant GNNs. 2. The theoretical analysis is thorough and provides valuable insights into the limitations of low-degree representations for symmetric structures. 3. The proposed HEGNN model offers a promising approach to incorporating high-degree representations while maintaining computational efficiency. 4. The experimental results on symmetric toy datasets align well with the theoretical predictions, providing empirical support for the main claims. 5. The paper includes a good discussion of the trade-offs between expressivity and efficiency in equivariant GNN models.

Weaknesses

1. The experimental comparisons on the N-body system are limited and use different splits and variations compared to existing literature. This makes it difficult to directly compare HEGNN's performance to state-of-the-art methods. Including comparisons with more recent baselines (e.g., ClofNet, GCPNet, SaVeNet) and using standardized benchmarks would strengthen the empirical evaluation. 2. The experiments are primarily focused on predicting future positions of particles/atoms. While this is a relevant task, it may not fully demonstrate the necessity or advantages of high-degree representations across a wider range of equivariant GNN applications. Additional experiments on different tasks or domains could provide more comprehensive evidence to support the paper's claims. 3. Some relevant baselines are missing from the comparisons. For example, GMN-L is not included in the MD17 dataset experiments, despite outperforming the proposed method on several targets. Similarly, GMN is missing from the N-body systems experiment. Including these baselines would provide a more complete picture of HEGNN's performance relative to existing methods. 4. The paper could benefit from a more extensive discussion of the computational trade-offs involved in using high-degree representations. While the authors mention that HEGNN is more efficient than traditional high-degree models, a more detailed analysis of the computational costs and scaling behavior would be valuable.

Questions

1. Can the authors conduct additional experiments with different tasks or application domains to further support the claims presented in the paper? This could help demonstrate the broader applicability and necessity of high-degree representations in equivariant GNNs. 2. How does HEGNN compare to more recent state-of-the-art methods on standardized N-body system benchmarks, such as those used in the ClofNet paper and subsequent works? 3. Can the authors provide a more detailed analysis of the computational efficiency of HEGNN compared to other high-degree models and EGNN? This could include training times, memory usage, and scaling behavior with respect to the number of particles and degree of representations. 4. Have the authors explored the performance of HEGNN on tasks beyond position prediction, such as force field prediction or other molecular property predictions? This could help strengthen the argument for the necessity of high-degree representations.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors acknowledge that their current experiments are mainly limited to testing on small molecules and have not been verified on large-scale molecules or large-scale physical systems. It remains to be verified whether HEGNN is effective on large-scale geometric graph datasets.

Summary

This paper studies the necessity of higher-degree features in geometric graph neural networks (i.e. graph neural networks processing data embedded in three-dimensional space), focusing on the ability to recognize data with non-trivial rotational inner-symmetry such as k-folds and regular polygons, and based on the findings, proposes and validates an extension of the E(n)-GNN architecture. Theoretically, the authors first state in Theorem 3.4 that any O(3)-equivariant function on an inner-symmetric graph must produce an output which has an identical inner-symmetry. This immediately puts a restriction on the possible space of outputs an equivariant function can produce, more specifically as the invariant subspace under the action of the inner-symmetry (Theorem 3.5). It follows that certain inner-symmetric inputs and certain choices of output spaces (i.e. degrees $l$) leads to degenerate output space {0} (Theorem 3.6 and Table 1). Consequently, it follows that using a sufficiently high-degree (even) features leads to non-degenerate output spaces of layers, hence be able to encode e.g. orientations of the input. The authors empirically validate the findings on expressive power in Section 5.1, and then demonstrate good performance of the proposed model based on high-degree features and invariant scalarization on n-body for n <= 100 and MD17 datasets.

Strengths

- S1. The paper is well-written and easy to follow. - S2. The claims on expressive power regarding symmetric inputs in Section 3.3 is correct, and is original as far as I can confirm; the closest related work is [1], and the authors have clarified the differences of their approach which seems technically correct. This claim is verified by the synthetic experiments in Section 5.1. - S3. The model proposed by the authors based on the theoretical claims is original as far as I am aware, and it seems useful in practice (Section 5.2-5.3), both in terms of performance and efficiency (slightly more costly than EGNN, but still cheaper than other networks involving higher-degree tensors such as TFN; but it is unclear how the approach compares to MACE, see W4 in the Weaknesses section). In particular, the fact that the model performs well on 100-body simulation task is interesting. [1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024)

Weaknesses

- W1. While technically different from [1], this paper conveys a similar message: the use of higher-degree tensors in geometric neural networks is necessary to obtain higher expressive powers which is necessary for certain problems. Given this, one may argue that the GWL hierarchy in [1] is more general (as it handles general non-symmetric inputs as well). Furthermore, inputs with rotational inner-symmetry have been investigated in Section 5.2 of [1] to argue in favor of the higher-degree tensor features, precisely using the task of encoding rotations of the input; while the types of inner-symmetries considered in this work is more general (Table 1), the finding is fundamentally not very different, which can be understood as a weakness of the paper. - W2. It was not entirely clear to me why the capability of encoding rotations of inner-symmetric inputs would be important for practical learning tasks, although the experiments seem to imply so. For example, on n-body systems, I believe it is very unlikely that one would encounter symmetric inputs including the ones given in Table 1, due to symmetry-breaking factors such as numerical imprecision, simulation errors, and noise. I have similar concerns regarding chemical structures (e.g. MD17), as slight perturbations to the atomic coordinates are sufficient to eliminate the inner-symmetries (if they exist) and hence not experience the problems given in Theorem 3.5 and 3.6. This implies a gap between the theoretical arguments on the expressive power given in Section 3, and empirical results given in Sections 5.2 and 5.3. - W3. While Theorem 4.1 says that the proposed architecture can recover the information of all angles between each pair of edges, it does not directly address the original problem given in Theorem 3.5 and 3.6. - W4. I am not sure why MACE [2], while also using higher-degree features and already used for experiments in Tables 1 and 2, is not included for comparison in Tables 3 and 4. [1] Joshi et al. On the Expressive Power of Geometric Graph Neural Networks (2024) [2] Batatia et al. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2023)

Questions

Please see the Weaknesses section.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations in the checklist, but not in the main text. I encourage the reader to move the discussions to a separate section in the main text or Appendix.

Summary

The paper studies the benefit of using higher order steerable features in geometric GNNs and theoretically identifies classes of symmetric geometric graphs where methods using only order-1 (or low order) features are guaranteed to fail. With this in mind, the authors propose a simple and efficient way to integrate higher order features in the EGNN architecture.

Strengths

Clearly written and well motivated paper. The proposed architecture seems a valid alternative to existing steerable architectures, enabling the use of higher order features with reduced computational complexity. The paper also includes a few interesting theoretical insights about the failure cases of EGNN.

Weaknesses

While the overall novelty is a bit limited (the message that higher order features are necessary for expressiveness has been theoretically and empirically explored in a number of previous works in the literature), the theoretical derivation of the failure cases and the proposed architecture can be useful contributions to the community. See questions below.

Questions

Eq 7, 8: do I understand correctly that this kind of model can not pass any information between features $\tilde{v}^{(l)}$ of different orders $(l)$ beyond the invariant quantities used to compute $m_{ij}$? Then, that essentially means that, after the initial features are extracted via the first convolution with spherical harmonics in Eq. 5, *the rest of model* is invariant to independent rotations by O(3) of each feature $\tilde{v}^{(l)}$, right? This seems to suggest there should be a gap in performance with models like TFN or SEGNN, which instead allow for different frequencies (features of different orders) to interact inside the model. Can you elaborate on this? Sec 5.2: it seems important to compare with SEGNN [21] since that work was also motivated by the idea of adapting EGNN to support higher order steerable features, if I remember correctly. Table 3: I would expect the model size to scale quadratically with the maximum frequency $L$ since $\sum_{l=0}^L (2l+1) = (L+1)^2$, but the runtimes in table 3 do not show this pattern. Why is the case? Table 4: what are the SOTA on these datasets? It would be good to include the performance achieved by some previous works to give an idea of whether these are competitive results or not Why introducing Theorem 3.5 when Theorem 3.6 seems more general and practical? Also, I think these theorems hold for any compact (not necessarily finite) subgroups by using the Peter-Weyl theorem. Line 226: is this a more efficient way to implement this fairly sparse operation?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

Some limitations are addressed in the supplementary material. For completeness, I think the authors could comment on the possible gap in expressiveness with other steerable methods like TFN and SEGNN which allow for features of different orders to interact (see my first question above).

Post rebuttal

I thank the authors for their responses and the additional experiments provided. I strongly encourage them to update their manuscript in several parts as per the reviewers' suggestions (e.g. extra explanations to improve clarity, clearer comparison with CG tensor product to illustrate their technical contribution, additional experiments suggested by reviewer vhjv, computational complexity/runtime discussion suggested by reviewer op9c). I maintain my initial score and positive evaluation of this paper and recommend acceptance.

Rebuttal Acknowledgment

Dear Reviewers and Area Chairs, We would like to extend our heartfelt gratitude for your dedicated efforts, insightful feedback, and constructive suggestions. Your nice comments and thoughtful suggestions are truly commendable. Through our discussions and the reviewers' responses, we convice we have addressed the major concerns raised by reviewers. Our work received positive comments from all reviewers both before and after the rebuttal, and now there are two reviewers have given clear acceptance recommendations (Reviewer 7wT3 & 3KcU). This outcome has greatly benefited and encouraged us, and we would like to thank all of you for your valuable support! Our theory strictly proved the necessity of high-degree steerable features, while our novel framework, HEGNN, proposed a promising method based on scalarization-trick to introduce high-degree steerable features with both efficacy and efficiency. We have made our complete code and training details publicly available, ensuring transparency and reproducibility. The rebuttal points raised during this process will be thoughtfully incorporated into the final version. Once again, we express our sincere appreciation for your time and dedication in reviewing our work. Your insightful input has significantly contributed to the refinement of our manuscript. Thank you!