Implicit Convolutional Kernels for Steerable CNNs

Summary

This paper tackles one problem of Steerable CNN: one needs to analytically solve a group G-specific equivariant constraint (eq 2 in the paper) in order to obtain the basics for the kernel. The authors propose to avoid this analytical solution by using a G-equivariant MLP to parameterize a G-steerable kernel basis (hence called implicit kernel). By doing so, they obtain a framework that allows one to achieve equivariance to various groups even when an analytical solution for the kernel basis is not know. The authors provide a theoretical proof that MLP equivariance is sufficient for building an equivariant steerable convolutional layer and empirically show that the suggested framework is useful in practice. Specifically, they show the relevance of the model being able to adapt to smaller G < O(n) (as opposed to assuming O(3) as in SEGNN) using the N-Body simulation system. They show the generalizability of their method on the ModelNet-40 dataset where the proposed technique works better than Standard Steerable CNNs when the analytical solution for the basis is not available. And finally they show the flexibility of their method that allows one to introduce additional constraints (such as bonds between molecules or atoms) on QM9 dataset. In both the last two tasks the authors compare their method with an extensive list of SOTA methods.

Strengths

- The paper tackles an important limitation of steerable CCNs. - The proposed solution is shown to be effective, at least when the analytical solution is unknown. Hence constitutes a detectable and positive improvement for the community. - The paper reads well.

Weaknesses

In my opinion there is an opportunity to strengthen the paper with a bit more empirical analysis. Since the kernel is learnt, it might be insightful to know to which extent the learnt solution is equivariant. Currently accuracy on downstream tasks is used as a proxy for that, but a specific analysis on equivariance violation could further enhance the paper. For example, is there a relation with the MLP size and the ability to learn the equivariance? Etc... Some parts could be more clear or more details could be provided. There are missing details about some experimental tasks, or how the model size used compares to the SOTA, etc… Please see the questions below for a more concrete list. The paper is not accessible to the wide ML community. Specifically the paper makes a lot of assumptions about the reader’s knowledge. I don’t think this is a reason for rejection but it would be great to make an effort, whenever possible, to make the paper more accessible. Just a few examples but there are many through the paper: - not all readers might be familiar with the concept of natural action (line 82). I’d recommend to clarify it. - "matrix containing the Clebsch-Gordan coefficients". Could explain what they are or add a reference. - Some of the notation used could be clarified, for example not everyone is familiar with the outer semidirect product used. It would be great to define it. **Minor notes:** - Definition 3: I’ would consider adding “for all g in G in x in X” - Line 289 “it’s” ==> it is (to be consistent with the rest of the paper) - Line 314 refers to Table 5.3 but I think it might be Table 1? - In the Bibliography the name of conferences are not consistent: sometimes it uses the full name of the conference and the abbreviation, some other just the abbreviation, other again just the full name.

Questions

I think the paper is good. There are opportunities to strengthen it by addressing some of the weaknesses mentioned above and by addressing the questions here. - Do the authors think it’s useful and possible to provide the empirical analysis mentioned in the weaknesses. - Can the author make an effort to make the paper more accessible? - In section 2 the authors write “Trivial representation: all elements of G act on V as the identity mapping of V , i.e. ⇢(g) = I. “ This does not seem to satisfy the definition 2, specifically this mapping does not seem to be invertible. Is this statement correct? - Can you describe maybe with an example how you would build the G-equivariant MLP? Section 3.4 would benefit from a more concrete explanation (even if in the appendix). Even better if the authors would consider to publicly releasing the code. - “We choose the model’s hyperparameters to have a similar parameter budget to the SEGNN”. Can one be more specific? How many parameters for example were trained in SEGNN compared to the proposed model? - In the ModelNet-40 experiment there is a sampling step involved. The Figure reports results with error bars using 5 runs but it’s unclear if the 5 runs are with different initial seeds or different sampling. Also is 5 runs sufficient? - In Figure 2: Do the authors have any intuition about why there is such a large STD for G=M but not for other groups? - 5.3: The task for this experiment could be explained better. The dataset is clear but what is the task? - QM9 task: “Both models we reported here have approx. 2 · 10^6” How does this compare to other models? - For the QM9 tasks more details about the meaning of the regressed energy values would help. Not everyone if familiar with this dataset and task. **Appendix B2** - For the N-Body “The run-time, on average, was 5 min. “ On which machine? And what is “run-time”? Per Single point? Per Bach? For the whole training? - For ModelNet-40 the machine is specified but the what is ”Runtime” should still be clarified.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The papers states the limitations in within the results discussion. For example
: - ”The only statistically significant negative difference is presented for G = SO(3), for which a tailored and hence optimal kernel basis is already available.” or - “Although we do not outperform more task-specific and involved approaches such as PointMLP [31],” or - "For the remaining energy variables (G, H, U, U0, ZPVE) and R2, our model significantly falls behind the task-specific benchmark approaches such as DimeNet or PaiNN.” Additionally there is a limitation section in Appendix C. I would encourage the authors to bring the limitation section from the appendix into the paper and summarize the limitations found during all the experiments in the same section.

Summary

This work considers the setting of steerable networks, in which a particular kind of group-equivariant (alternatively, G-steerable) kernel is translationally convolved with an input vector (or matrix) field. The group-specific design of such a network focuses on the derivation of the group-equivariant kernel. Although prior work laid out a procedure for deriving an analytic, orthogonal basis of functions whose span encompasses all such G-steerable kernels, this paper presents a conceptually simpler method: parametrizing the kernel with an equivariant map. This method is flexible and allows conditioning the kernel on other invariant or equivariant attributes. They test their method on datasets of point cloud classification, synthetic N-body simulations, and molecules (QM9).

Strengths

Originality: Using an equivariant network to parametrize a group-steerable kernel is a natural and elegant idea. Although implicit kernels have been used in other settings, their application to steerable kernels is novel and well-founded. Conceptually, it is simpler than previous analytical approaches to deriving explicit, orthogonal bases. Clarity: The proposal is clearly described, and the paper provides a helpful consolidation of related work overall. Quality: The proposal is indeed equivariant, and experiments show good results over a sufficiently wide range of settings. Significance: Group-steerable networks are widely used, and this formulation seems to apply to all compact groups of interest.

Weaknesses

1. It seems to me that the vast majority of application problems involving subgroups of $E(3)$ acting on $\mathbb{R}^3$ simply involve $SO(3)$ or $O(3)$. In that sense, the scope of this paper may be limited. Nonetheless, this does not preclude other groups of interest from arising in future applications, but the paper could be strengthened by providing additional motivating examples. 2. The tradeoffs between this method and the analytical method of [7] are subtle, and indeed they require many of the same tools (for example, knowing the irrep structure of the group of interest, so that one can construct equivariant functions over the input space, even if they are not required to be orthogonal). As noted in the questions, I do not fully understand the benefits of this method as compared to [7], although it is clearly more general and simpler to articulate.

Questions

1. My biggest question is that I do not understand the second paragraph of Section 3.2, comparing this approach with that of [7]. The paragraph seems to claim that the basis produced by [7] might be too large, but I thought [7] would produce an orthogonal equivariant basis — and if it is orthogonal, how can the “size” of the basis be suboptimal (as it is determined the dimension of the vector space of steerable functions)? 2. On a related note, both this paper and [7] seem to require knowledge of the irreps of the group of interest. Can the authors elaborate on which of the ingredients (a) through (d) listed at the bottom of page 5 of [7], are troublesome? Section 3.2 seems to imply not that the subgroup method for obtaining a steerable basis is computationally intensive, but that it produces a lower quality output — is that the case? 3. If already provided with a steerable basis in the style of [7], could one still “condition” on physical info (at least invariants) by just storing a bank of coefficients, and choosing different coefficients based on endpoint features? If so, this could make more sense as a baseline experiment. Of course, such a method may not choose coefficients smoothly as a function of the endpoint features, which is one potential benefit of the authors’ proposed conditioning method. 4. To clarify, if you had a G-equivariant MLP, is the primary barrier to using this G-MLP to derive a G-steerable orthogonal basis the orthogonalization? In other words, one could set random weights to the G-MLP, and then try to orthogonalized to get a basis of equivariant functions? (I think the problem is this: orthogonalization can be done via Gram-Schmidt, but only if one can compute inner products exactly — with $\mathbb{R}^3$ as the base space, it does not seem like one could do this exactly. However, I would be happy to hear the authors’ thoughts on this, as if there were an easy way to reduce a G-equivariant MLP directly to a steerable basis, it would perhaps weaken the argument that their proposed method is easier than computing an explicit G-steerable basis.) 5. In Section 5.2, why is the baseline an $O(3)$-equivariant net? This is an unfair comparison for a problem that is not $O(3)$-equivariant, since such a network can never express the ground truth solution. Can the authors report performance relative to a generic net instead (e.g. only translation invariant), or an approximately equivariant network? 6. What do the authors hypothesize causes the difference in performance shown in Figure 3? For example, is it conditioning on radial vectors? As a quick note, here is one other piece of slightly related work at the intersection of equivariance and implicit convolutional kernels: “Relaxing equivariance constraints with non-stationary continuous filters” by Ouderaa et al 2022.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

There are not potential negative societal impacts.

Summary

The paper proposed implicit neural representations via MLPs to parameterize G-steerable kernels. Additionally, the performance of the approach is extensively empirically tested against alternative methods on multiple tasks.

Strengths

- Conceptually In many cases, steerable group equivariant neural networks yield more accurate equivariance in practice compared to counterparts with regular representations that use sampling. Implicit kernels have shown to be beneficial in the space of regular group equivariant networks but have not yet been explored in the context of steerable equivariance. This paper demonstrates how steerable equivariant networks can also be extended to use implicit kernels. - Scientifically The paper extensively validates the method on actual datasets in which more complex symmetries play an important role. The paper extensively validates the approach on both graph structures and dynamical systems, demonstrating relevance in real-world applications and also includes extensive comparison with existing baselines.

Weaknesses

- Implicit kernels do not alleviate the need to solve Eq.2 analytically. The method relies on steerable kernels, which require analytically solving Eq. 2 for specific group G of interest. Implicit kernels do not alleviate this, as the constraint of Eq. 2 must still be solved in order to build G-equivariant MLPs. Although these are shortcomings of steerable equivariance and not of the implicit kernels, it should be stressed that implicit kernels do not mitigate these shortcomings. - High computational complexity and memory cost The method only seems to increase computational and memory complexity over already more expensive steerable kernels. - No benefit with increased depth of implicit kernels The paper relies on using multi-layer MLPs as implicit kernels but notices that deeper MLPs do not necessarily lead to improved performance (App. B.3 and App. B.4). The paper does argue that the increased flexibility that implicit kernels offer may benefit long-range dependencies but does not conduct any experiments on this. It would be interesting to demonstrate the benefit of implicit kernels on tasks where deeper kernels help. - Class of implicit functions Is there a reason why only multi-layer perceptrons are considered to parameterise implicit functions? Since the depth of implicit functions has not been found to be beneficial, it could also be interesting to consider different simpler non-linear bases. Even more so, because they appear to be effective in the context of regular group equivariant networks (e.g. sinusoidal/fourier [1], b-splines [2] or simple radial basis features [3]). Or is there a reason why this would not be applicable or less interesting in this setting? [1] Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473. [2] Bekkers, Erik J. "B-spline cnns on lie groups." arXiv preprint arXiv:1909.12057 (2019). [3] van der Ouderaa, Tycho FA, and Mark van der Wilk. "Sparse Convolutions on Lie Groups." NeurIPS Workshop on Symmetry and Geometry in Neural Representations. PMLR, 2023.

Questions

a) Am I correct in the assessment that the method does not alleviate the need to solve Eq. 2 analytically and thus does not directly improve the efficiency of steerable kernels in that regard? b) What are computational runtimes? (and what is the explicit overhead of implicit kernels) c) What are memory costs? (and what is the explicit overhead of implicit kernels)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper extensively compares (externally) with alternative approaches. However, the (intrinsic) analysis of how different types of implicit kernels may affect performance remains limited. One of the main benefits of implicit kernels is the increased flexibility it provides when designing the model. It would therefore be very interesting to get a better idea of how different choices in the functional form of the implicit kernels affect the output or model performance. This point becomes especially relevant since the paper includes negative results on deeper implicit kernels (which nevertheless are interesting findings), as this would help demonstrate how implicit kernels can be beneficial in practice.

Summary

This work proposes a novel method for achieving a G-equivariant neural network by implicitly parameterizing the steerable filters by G-equivariant MLP instead of learning them with a steerable basis function. The proposed frame is flexible, and the implicit kernel can also consider the problem context by expanding input to the kernel with problem-related parameters. The work provides theoretical justification for the proposed method and shows the model's effectiveness via empirical study.

Strengths

1. The work proposes a novel method to achieve G-equivariant by implicit parameterization, which provides more flexibility to the G-equivariant model. And empirically shows the benefit of the flexible kernel that can consider the additional problem-dependent features. 2. The text is easy to follow and clearly written

Weaknesses

1. Evaluation: The empirical evaluations are performed on N-body simulations, ModelNet, and the QM9 dataset. Which is different from the previous works [1,2,3]. And this makes it difficult for readers to make a direct comparison between the approaches. And raises the question of whether the performance of the approaches is significantly dependent on the chosen problem. 2. The proposed method utilized G-equivariant MLP for implicit parameterization of the kernel. But the rationale for choosing an implicit steerable kernel over EMLP[4] is not addressed nor included in the evaluations. 1.Steerable CNNs 2.3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data 3.Clebsch–Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network 4.A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups

Questions

1. The G-equivariant MLPs also require us to solve constraints [1]. And we need to do the same for the steerable methods. How are these two different? 2. The result of Table 1: What is the rationale for choosing a problem where we have non-G-equivariant models outperforming G-equivariant methods? 3. Line: 105, equation 2 seems a little out of context. Even though the relevant work is cited and a little background on how we reach that constraint will make the work self-contained. 1.A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Group

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Thanks for considering my suggestions and addressing my questions

I would like to thanks the authors for answering my questions. I am satisfied with the answers and I believe that if the authors will include the modifications suggested in the manuscript the paper will be stronger. The only two comments (none strictly needed for a weak acceptance but something for the authors to consider) that I would like to add are: 1. On the first weakness: as the authors mentioned, there is a gap between the theory and the practical implementation. What I think would add value is to test to which extent those design choices ( e.g., non-linearity) have loosen the exact equivariance. 2. On the statement "Five runs is a typical choice in deep learning experiments". I believe this is an arguably bad choice, especially in a case (like this one) where there are two sources of stochasticity (random seed and sampling step). I believe that a more statistically significant analysis would have made the empirical evaluation stronger (but again not needed for weak acceptance).

Response

I thank the authors for their response.

Thank you for the response

Thank you to the authors for their thoughtful response. I am inclined to raise my score, but I have just a few remaining questions and comments. 1. Thank you for the new experiment with a non-equivariant baseline. I am satisfied with these results. 2. Unless I am mistaken, Lemma 1 is a basic linear algebraic rearrangement of Equation 2. Solving one seems equivalent to solving the other. Therefore, the response to reviewer fqKN’s first question does not make sense to me. 3. Do the advantages of this work hinge on the assumption that the equivariant MLP is a universal approximator? 4. In the case of SO(3), the spherical harmonics weighted by radial basis functions provide a G-steerable basis for SO(3) [3D steerable CNNs, Weiler et al 2018]. To what extent (if any) does this phenomenon generalize to other groups? I find this important to check, because I would like to clarify whether knowing an analytic form for the irreducible representations of a group can provide a G-steerable basis.