We introduce Clifford Group Equivariant Neural Networks: a novel approach for constructing $\mathrm{O}(n)$- and $\mathrm{E}(n)$-equivariant models. We identify and study the $\textit{Clifford group}$, a subgroup inside the Clifford algebra tailored to achieve several favorable properties. Primarily, the group's action forms an orthogonal automorphism that extends beyond the typical vector space to the entire Clifford algebra while respecting the multivector grading. This leads to several non-equivalent subrepresentations corresponding to the multivector decomposition. Furthermore, we prove that the action respects not just the vector space structure of the Clifford algebra but also its multiplicative structure, i.e., the geometric product. These findings imply that every polynomial in multivectors, An advantage worth mentioning is that we obtain expressive layers that can elegantly generalize to inner-product spaces of any dimension. We demonstrate, notably from a single core implementation, state-of-the-art performance on several distinct tasks, including a three-dimensional $n$-body experiment, a four-dimensional Lorentz-equivariant high-energy physics experiment, and a five-dimensional convex hull experiment.
Paper
Similar papers
Peer review
Summary
This paper presents a novel approach for creating E(n)-equivariant models based on Clifford (geometric) algebras. The authors identify the Clifford group and its action. By extending this action to the entire Clifford algebra, remarkable properties can be obtained enabling the construction of equivariant maps from the Clifford algebra to itself, as well as other equivariant operations, such as grade projection. Using these, an equivariant Clifford group neural network operating on vector fields can be constructed, capable of a more accurate and nuanced interpretation of the underlying structures than the baseline scalar-field networks. The proposed method exhibits application versatility across different tasks and dimensionalities of the space.
Strengths
S1. The high quality of writing and presentation are two strong features of the paper. The authors succeeded in finding a good balance between the amount of technical detail and the narration clarity. S2. The paper presents novel theoretical results based on Clifford algebras, enabling an original way of constructing equivariant neural networks, which is a great contribution to the field. S3. Experimental validation is versatile and, to a large extent, convincing.
Weaknesses
W1. Missing details in the presentation of the proposed method, including: (See the questions section for more information.) - translation equivariance (Q1) - invariant prediction computation (Q2), - the architecture of the proposed networks in the experiments (Q4). W2. Incomplete comparison with other methods: - The support of the claim that the scalar-feature methods are not able to extract some essential geometric properties from input data needs to be elaborated on (Q3). - Discussion on the complexity of the proposed method vs the baselines is missing (Q4). - Some training details are missing (Q5). Addressing these weaknesses will improve the presentation and clarify the support of certain claims.
Questions
Q1. Translations: - The authors claim to present an E(n)-equivariant method. - As discussed in Section 3.2, equivariance w.r.t. the Clifford group implies equivariance w.r.t. the orthogonal group O(n). It is, however, unclear how **translation** equivariance is attained. In fact, all the presented experiments involve only the orthogonal group transformations: in the $n$-body experiment, the authors use mean-subtracted positions of the particles, thereby removing the effect of translation already on the input level. - Could the authors clarify how translations can be handled using the proposed layers? Q2. How are O(n)-**invariant** predictions obtained with the proposed method? Q3. The claim of better underlying geometric properties capturing is meant to be supported by the *signed-volume experiment*. I would like for the authors to elaborate on the claim itself and clarify the presented experimental support. - How do the authors define *covariance*? - Some of the considered scalar-feature methods extract equivariant features and compute the invariant features from them to perform classification/segmentation requiring this. For instance, in the theory of the VN method [34], there's nothing that restricts the features to only be equivariant under rotations and not reflections. In fact, the invariant features are obtained as the inner product of equivariant features, which also cancels the effect of reflections thus making the method produce O(3)-invariant predictions. This makes the VN method unable to distinguish a tetrahedron and its reflected copy (and thus, their signed volumes will not be distinguished either) by the design of the invariant computation block only, even though the method extracts equivariant features. This should be taken into account in the comparison in Section 5.1. - Could the authors provide more details of the experimental setup? I want to see the number of samples, the ratio of positive/negative volumes, and the success rates of the predictions given positive/negative volumes for the proposed method and the baselines. Q4. Could the authors provide details of their method architecture used in the experiments in Sections 5.1-2? How does the complexity of the method compare to the baselines? Q5. Is the training setup the same for all the methods in the experiments? I.e., were the training hyperparameters optimized for the proposed method and used for others?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The limitations and broader impact are adequately addressed by the authors.
Summary
I like to admit that I am not familiar with this area of work. I try my best to review the key contributions of this work. My review is more of clarification than critic. The theory part of the paper lies mainly in section 3.1 starting with Eq.(5). The computation part of the paper is given in section 4. The authors introduce two kinds of layers. Linear Layers and Geometric Product Layers. Linear layer mix the input channels of the same grade (Eq.(11)). Interaction layer takes the geometric product of the i and j grade of x1, x2 and project to the k-th grade. Several experiments were performed using this neural network. I will discuss more about the experiments in later part of this review.
Strengths
1. The paper explore a different approach to deep learning. This paper helps in adding a new dimension in the field of deep learning. 2. The authors provided a huge amount of background and foundations in the supplementary materials. This is important because I presume relatively few readers are familiar with Clifford algebra. 3. Although I cannot fully understand the theoretical part of this work. It seems to me the authors try to substantiate their work with theories.
Weaknesses
1. The strength of this paper may also be "its weakness", readers without background on Clifford algebra may struggle to pick up this area of work. It will be good if the authors follow up with a good tutorial. Perhaps a series of online learning materials and provide its link in this paper. For example, make an archive tutorial paper (or online video) and those locations are cited in this paper. The supplementary materials of this paper is good. but it is insufficient to bring people outside of this field up to the level to use this paper. 2. The output of the neural network is given by Eq.(11) and Eq.(13). I understand that the output is also an element of the Clifford algebra. I struggle to understand how this output is link to the use cases in the experiments. It will be good if the authors help the readers by describing the experiments in detail. Give a step-by-step walk through of : 2a: how the data is being prepared and in what format 2b: how the data is being feed into the neural network 2c: how to link the k-th grade of x to the numerical output. 2d: perhaps walk through with a simple example, like one example using complex numbers. I explain 2c a bit more here. For a complex number c1 = 2+3i, the zeroth grade is 3 (or the 1 element). The first grade is 2 (or i). If the author can guide the reader through on how to input c1 into the linear layer and explain what is the output, that will help. Similarly, the author may also give another example c1=1-2i and c2=4+I on how to feed c1 and c2 into Eq.(12) and Eq.(13).
Questions
How to prepare data to feed into the neural network? Perhaps the authors can give a simple example(s) of processing complex numbers and/or quaternions.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Readers without a background of abstract algebra or Clifford algebra may find it hard to understand this paper.
Summary
In this submission the authors introduce an equivariant neural network architecture building on the Clifford algebra (defined by a vector space and a quadratic form). They remind that this algebra admits an orthogonal decomposition over sub-vector spaces called grade $m$ where $m$ is the number of basis elements. This is referred as the 'multivector grading of the Clifford algebra'. They identify a (Clifford) subgroup and its (adjusted twisted conjugation) action which not only acts on the original vector space but also on the Clifford algebra whilst satisfy many properties including multiplicatively w.r.t. the geometric product. They show that equivariance w.r.t. this a Clifford group acting on the Clifford algebra implies equivariance w.r.t. the orthogonal group acting on the Clifford algebra. Then they propose several layers, including a linear layer and a geometric product layer which they show to be equivariant w.r.t. the action of the Clifford group. Then they empirically assess their proposed approach on synthetic tasks with $O(3)$, $O(5)$ or $E(3)$ equivariance and show that it overperform scalar based methods but also steerable NNs. Finally, they tackle the task of identifying top quarks from high-energy jets produced in particle collisions at CERN, and show that they are able to perform as well as recent specialised neural networks.
Strengths
- This is a nice paper that builds on this recent research avenue of equivariant neural networks based on the Clifford/geometric algebra. - In find particularly appealing the fact that the methodology applies to any dimension, and more generally than $O(n)$ equivariance via the choice of quadratic form $\mathfrak{q}$ as per Section 5.4 (if I understood correctly). - Additionally, it appears that the method is empirically able to fit well a variety of functions and able to generalise. - The paper is pretty well written given the space constraint, although I give some suggestion on improvements below.
Weaknesses
- Likely due to space constraint, yet the methodology would deserve more discussion. - For instance, how is the data embedded into the algebra (e.g. for higher order quantities such as an inertia tensor)? - How is the quadratic form $\mathfrak{q}$ chosen (e.g. is it the Minkowski product for the Jet tagging experiment)? - I also believe that the linear and geometric layers could benefit from an illustrative diagram. Perhaps it may be useful to draw parallel with tensor product and irreducible representations for readers familiar with this language. - What's more the main paper is currently lacking a discussion on scalability of the method, and how this compares to Clebsch-Gordan tensor product approaches, although I understand that the space is limited. - Another points which I find slightly frustrating is the lack of ablation study, as generally it is hard to identify why this method overperform tensor product / irreps approaches. Is it more parameter efficient? Does it generalise better? Does it yield a better optimisation landscape? Is it computationally faster? Obviously fully answering this questions would deserve a full paper on its own, yet some elements of answer would be nice :)
Questions
- line 62: 'our method readily generalizes to orthogonal groups regardless of the dimension'. Doesn't the method proposed in [Cesa et al. 2021] also work for any dimension $n$? - Eq 1: What is the intuition behind the independence of the choice of orthogonal basis? Is it because by combining elements from one basis (via the geometric product) one could obtain the second basis? How come the limiting factor is really the number $m$ of such basis elements? - line 232: Worth saying a bit more about the 'layer-wide normalization' proposed in [70]? - line 235: 'However, we still require activation functions after linear layers'. Does this statement come from practical requirements for the method to work well, or from some universal approximation theorem with necessary non-linearity after the linear layer? - Section 5.1: Is the 'signed volume' a pseudo-scalar? - One of the advantage of this method that it readily works whatever the input space dimension, yet what is the scalability of the method with respect to the dimension of the space? - Section 5.2: What are the reasons for EMLP underperforming? It is theoretically a universal approximator. - Section 5: Do all methods have the same number of parameters? or same compute time? A PROGRAM TO BUILD E(n)-EQUIVARIANT STEERABLE CNNS, Cesa, Gabriele and Lang, Leon and Weiler, Maurice, 2021
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
2 fair
Contribution
3 good
Limitations
- 'CGENNs induce a (non-prohibitive) degree of computational overhead'. Where does this overhead come from? What specific operation would benefit from having a custom GPU kernel? More generally it would be really nice to discuss the scalability of the method. In particular vs other 'steerable' methods which rely on parametrised tensor product via Clebsch-Gordan coefficients.
Summary
The paper introduces a new method for constructing E(n)-equivariant networks called Clifford Group Equivariant Neural Networks (CGENNs), based on an adjusted definition of the Clifford group, a subgroup of the Clifford algebra. The researchers have shown that the group’s action forms an orthogonal automorphism that extends beyond the typical vector space to the entire Clifford algebra, respecting its multivector grading and multiplicative structure. This leads to non-equivalent subrepresentations corresponding to the multivector decomposition and allows parameterizing equivariant neural network layers. CGENNs operate directly on a vector basis and can be generalized to any dimension. Incorporating group equivariance in neural networks has proved useful in many areas, including modeling the dynamics of complex physical systems and studying or generating molecules, proteins, and crystals. This paper demonstrates that the CGENNs can handle directional information more accurately and nuancedly than scalarization methods, operating directly in a vector basis rather than alternative basis representations. They can also transform higher-order features carrying vector-valued information, and readily generalize to orthogonal groups of any dimension or metric signature. The authors successfully demonstrated these advantages on various tasks, including three-dimensional n-body experiment, a four-dimensional Lorentz-equivariant high-energy physics experiment, and a five-dimensional convex hull experiment.
Strengths
**Originality** This paper has many strengths across multiple dimensions. First, it tackles the problem of building an Equivariant network using a fresh perspective that has not been, as far as I know, used in the literature. Thus the use of the Clifford Group is an exciting direction of technological endeavor. Certainly, the mathematics behind building the Linear, Geometric Product, and Nonlinearities is nontrivial but offers what appears to be greater modeling flexibility. **Quality** It is abundantly clear that immense care has been taken with the writing of this paper. This is substantiated by the large and extensive appendix which in itself can prove to be a useful reference outside of the main thesis of this paper. Furthermore, it is clear that a computational approach was taken to build the equivariant layers as I appreciated the authors that the authors acknowledged the naive $(n+1)^3$ interactions needed in their geometric layers which could be reduced by first applying a linear projection. **Clarity** The authors should be given credit for attempting to make this paper clear and accessible. Unfortunately, it is not outside for those who are likely already familiar with geometric algebras. Although the writing and the flow of information are still very clear if one pretends to know what the Clifford group is. **Significance** In this reviewer's opinion, this work is significant for several reasons. First, it allows for the creation of equivariant networks in a completely different manner and one that appears to be more flexible/expressive. This is due to the Clifford group allowing for non-equivalent subrepresentations. For example, Clifford group equivariant models can extract covariant quantities which cannot be extracted from scalarization methods. Furthermore, the early empirical results suggest that such an approach also leads to empirical benefits and beats current equivariant models which suggests that this is a ripe direction for continued investigation.
Weaknesses
This paper has a few areas for improvement that I highlight below. 1.) I thank the author for their efforts to make the paper accessible, but the appendix is not a very friendly introduction. For instance, in some places, the authors acknowledge that their definitions depart from existing ones in the literature. But in other places, they claim to follow the format of other works (e.g. C.1). This makes it hard to follow and trust what the authors state. 2. Perhaps what bothers me most is the fact that the motivation for using the Clifford algebra group is lost in all the technical jargon. The importance of being able to use multi-vectors in ML-specific problems is never really motivated in a convincing way. Even at present, I find it difficult to pinpoint exact non-toy applications where these tools might prove useful. Albeit there is no doubt the tools themselves are mathematically interesting. 3. There are parts of the paper where it becomes unclear which part of the theory is an original contribution of the authors versus something that is already present in the math literature on Clifford group/algebra. As far as I can tell the design of the adjusted twisted conjugation is novel and its extension is an original contribution of the authors (and the subsequent ML-specific layers). Can the authors tell me if I missed anything? 4. The experiments are quite preliminary. Perhaps this is by design as the authors note that the computational complexity can partially be resolved with better hardware-specific implementations. I would have loved to see the power of these methods on larger-scale equivariant problems. For instance, I am certain the authors are familiar with molecular dynamics simulations at a larger scale. These would already have the desired symmetries and established baselines to compare against. 5. The authors could improve the method section by providing an empirical investigation of the runtime complexity of their actual experiments in comparison to the baselines. Right now, a reader may intuit that the Clifford group equivariant networks are more computationally expensive but, exactly how much is not immediately clear.
Questions
1. One part that is not clear to me is how equivariance is preserved in the geometric layers if you use a linear map to first project down to $l$. This would appear to break equivariance. Can the authors please comment on this and how it doesn't break the desired symmetry? 2. A limitation of the steerable equivariant literature is that it is heavily reliant on the Wigner-D matrices for $O(3)$. It seems the current effort in the Clifford group also applies to the orthogonal and $E(n)$ group. Can this approach be used to model other continuous symmetries e.g. $SU(2)$ or is this not possible?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
4 excellent
Limitations
N/A
Summary
The paper develops a new kind of neural network model based on the Clifford group, which makes the neural network E(n)-equivariant. The model is tested on a number of datasets, showing that it is capable of recovering the desired structure from the data.
Strengths
This seems like an interesting and important direction. The results are convincing, in the sense that this approach "works." The method is backed up by nice theory.
Weaknesses
Although the intro and figure 1 at the beginning are helpful, I think the rest of the paper is hard to follow. There are some theoretical results, but then when it comes to the methodology, section 4, I am left wondering what is actually being computed. For example, what is involved in computing equation 11? it says the x_i are elements of Cl(V,q), ok but what is the input to this system? same for eqs. 12 and 13. And what is the objective function? For the first experiment on signed volumes, this seems like an interesting task, but it is not clear what is the form of the input data - a data cloud of x,y,z points? What is the input to the Clifford network? I am having a hard time picturing concretely what is going on here. More specifics would be helpful.
Questions
How would you compare to the following paper: https://proceedings.neurips.cc/paper/2021/file/f1b0775946bc0329b35b823b86eeb5f5-Paper.pdf. This paper develops a universal family of neural networks for O(d)-equivariant functions. This seems to stand in tension with the claims in the Clifford algebra paper that scalar feature-based methods cannot capture certain kinds of information. (I'm sure that some more thought could explain what's going on, but I feel like that's work the authors of the submission should do, rather than the reader.) Relatedly, I don't think I saw any statement about whether or not the Clifford algebra approach is universal (maybe I missed it?). Also, the authors end Section 3 by saying "equivariance w.r.t. the Clifford group [...] implies equivariance w.r.t. the orthogonal group," but this statement, by itself, still leaves a question open: Do there exist functions equivariant under O(d) but not the "Clifford group," which their method then couldn't learn?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
n/a
Re:Rebuttal
I thank the authors for their detailed rebuttal. I am mostly satisfied with their answers. I still contend that the Appendix is not an easy introduction to the subject as I might care for but given the scope and divide between the different communities it is still a welcome first step. As such, I'm upgrading my confidence to a 4 and maintaining my original score (7).
We are grateful for your acknowledgment of the improvements we've made in our rebuttal and are happy to hear that our efforts have enhanced your confidence in our work. Regarding the appendix, we understand and respect your perspective. It's always a balancing act to cater to various communities with varying familiarity with the subject. We'll continue to strive for clarity and accessibility in future iterations. Your feedback has been instrumental in this process.
I appreciate the authors' comprehensive rebuttal addressing all of my questions and thank them for that! I would further like to clarify the following items. 1. *Translation equivariance* Is the network configuration used in the $n$-body experiment translation-*equivariant*? I.e., do the authors add the subtracted point onto the prediction? I would recommend that the authors include the translation-equivariance clarification in the paper. Otherwise, it is not entirely clear how exactly the currently presented method is suitable for constructing *E(n)*-equivariant networks (line 2). 2. *Neural architecture* With reference to Table 1 in the rebuttal PDF and comparing the architectures for O(5)-volume (convex hulls) and O(5)-regression experiments, I wonder what causes the difference in the number of parameters. In both cases, the authors use a 4-layer feedforward network. However, for the (seemingly simpler) task of O(5)-regression where the input is 2 5D points (vs., I presume, larger input for convex-hull volume prediction), the number of parameters is one order of magnitude higher. - Is there a typo or how can the authors explain this otherwise? - Using a feed-forward neural structure in experiments 5.1-5.2, do the authors address the *permutation equi- and invariance* wrt the input points/particles? Or is this requirement relaxed? - Could the authors also clarify this for experiments 5.3-5.4?
Thank you for your thoughtful queries and for giving us the opportunity to further clarify our work. **Translation equivariance**: In the $n$-body experiment, we have indeed outlined the translation equivariance by mentioning, “The input consists of the mean-subtracted particle positions (to achieve translation equivariance) and…”. To make this clearer, we will explicitly mention in the revised version that we add the subtracted point back onto the prediction. We agree that emphasizing this point will provide better clarity on how our method achieves E(n)-equivariance, and we appreciate your recommendation in this regard. **Neural architecture**: The difference in parameter count between the O(5)-volume (convex hulls) and O(5)-regression experiments is primarily due to the different numbers of hidden dimensions used. The latter uses hidden dimensions of 128 (which is arguably overparameterized) compared to the former's 16 dimensions. For the regression experiment, we intentionally matched the number of parameters to those of the baselines to ensure a fair comparison. We'll ensure that such details are more explicitly mentioned in the revised manuscript. **Permutation invariance**: In our synthetic experiments, we relax the permutation invariance. Specifically: for the convex hull experiments, our goal was to test our raw feed-forward parameterizations in their most unadorned form against the baselines, among which some were also introduced as feed-forward architectures. **Dealing with permutation invariance in experiments 5.3 and 5.4**: We employ graph neural networks (GNNs) to address the permutation invariance for these experiments. Although our manuscript was limited in detailing this due to space constraints, we briefly touched upon using graph neural networks in 5.3. Delving deeper, these are straightforward message passing networks wherein both the message and update networks utilize our proposed layers. We acknowledge the omission of some of these details, especially regarding experiment 5.4, in our pursuit to fit the space constraints. We will rectify this by adding these details back in the revised version for a clearer understanding. We hope that these explanations successfully addressed your comments. We appreciate the time you spent on this review, which has resulted in an improved revision of the manuscript. If it aligns with your assessment, we'd be grateful if you'd consider promoting our score. As always, we remain at your disposal for any further questions or clarifications.
Thank you for thoroughly answering my queries! Based on the quality and content of your rebuttal addressing my concerns about the original manuscript, I am confidently raising your score.
I like to thank the author for answering my queries. I will work through the paper again. Really look forward to any new tutorial materials. I understand there is a deadline for rebuttal and focus of the author should be to discuss here. I look forward to the eventual tutorial materials on arxiv after the whole process of double blinded process is over.
response
Thanks for taking the time to address my questions and remarks, but also for providing additional visualisations! I acknowledge that it's tricky to strike a good balance between clarity and space constraint, and believe that the submission is doing a pretty good job already.
Decision
Accept (oral)