Noether's razor: Learning Conserved Quantities

Summary

This work proposes a novel way to parametrise Hamiltonians that are approximately invariant to a set of learnable symmetries with the aim of improving existing methods of learning Hamiltonians of dynamical systems. It does so by forcing Hamiltonian neural nets (HNN) to adhere to symmetries, just as in reality, through Noether's theorem. Specifically, they parametrise conserved quantities as quadratic functions for which the flow can be analytically solved and approximately integrate a non-invariant parametrisation of the Hamiltonian (given by an existing HNN) over this flow. The result of this integration is then known to be approximately invariant to the symmetries represented by the quadratic conserved quantities. To optimise their model, which produces an intractable marginal likelihood, the authors resort to variational inference as was done in [1]. In effect, they apply the methodology of [1] to a learnable set of conserved quantities and show experimentally that it allows them to learn the underlying symmetries of different dynamical systems. [1] van der Ouderaa, Tycho FA, and Mark van der Wilk. "Learning invariant weights in neural networks." Uncertainty in Artificial Intelligence. PMLR, 2022.

Strengths

1. The structure and writing of the paper is fairly good and makes for an easy-to-follow story. I appreciate the completeness of the preliminaries as it makes for a self-contained and clear paper. 2. The goal and main research question that the paper tackles is also very clear from the start: How to learn a representation of the Hamiltonian of a dynamical system that also adheres to symmetry, just like real Hamiltonians. The motivation as to why this is important is also clearly illustrated by the experiments, showing how representations without symmetry (regular HNNs) fail to model good Hamiltonians. 3. The evaluation of how well the learned conserved quantities relate to the true underlying symmetries is reasonably well done. Moreover, it does seem to indicate that the proposed method can learn good representations of symmetries despite the use of approximations.

Weaknesses

1. While the paper is well-structured and self-contained, it is not very clear which parts are background and which parts are the novel contributions. In particular, going through the related work, it seems the only real novelty is the parametrisation of symmetries of H via quadratic conserved quantities (Section 3.1). 2. The main weakness seems to be the novelty of the paper. Apart from providing a way to parametrise conserved quantities, the rest of the utilised methodology is very close to [1]. To be specific, Section 2 consists of the necessary preliminaries while Section 3.2 explains a well-known procedure to approximately symmetrise a non-invariant function [1]. Section 3.1 is surely novel, but it could be made clearer why quadratic conserved quantities were chosen, i.e. because they lead to analytical flows that are necessary in the symmetrisation. Finally, most of Section 4 and the corresponding appendices are essentially equivalent to the derivations given in Sections 3.6 and 3.7 of [1]. Even the same notation is (wrongly) used in the proof given in Appendix A.1: $E_{\tau} = E_{\tau}\sim E_{\prod_{s = 1}^S p(\tau)}$, which should probably be $E_{\tau} = E_{\prod_{s = 1}^S p(\tau)}$. 3. The proposed metrics to measure how similar the learned symmetries are to the ground truth symmetries make sense, but I have some doubts about the "parallelness" metric. Specifically, the use of the Euclidean norm can give a skewed perspective as it is not an ideal metric for higher-dimensional vectors [2]. Are there no established methods to measure closeness to symmetries? If not, then the proposed metrics are another contribution. If yes, then please cite the necessary literature. 4. The experimental evaluation is lacking variability metrics in general, making it very hard to grasp whether the differences are actually statistically significant and meaningful. Please include either standard error, standard deviation or at least 25-75 percent quantiles over a repeated set of independent runs. Secondly, I wonder why there aren't more baselines for the experiments, especially since the authors mention a couple of competitors [3, 4]. If they are not applicable, then this should be made more explicit as it can also help strengthen the proposed method. 5. I do not agree that the use of approximate Bayesian inference in the context of model selection for deep learning has not been studied/scaled to neural networks as stated in the conclusion (lines 359-360). I would argue that any application of variational inference to Bayesian neural networks (BNN) for example is always a form of model selection as it selects a simpler, yet performant approximation to the BNN within the provided variational family [5, 6]. [1] van der Ouderaa, Tycho FA, and Mark van der Wilk. "Learning invariant weights in neural networks." Uncertainty in Artificial Intelligence. PMLR, 2022. [2] Aggarwal, C. C., Hinneburg, A., & Keim, D. A. (2001). On the surprising behavior of distance metrics in high dimensional space. In Database theory—ICDT 2001: 8th international conference London, UK, January 4–6, 2001 proceedings 8 (pp. 420-434). Springer Berlin Heidelberg. [3] Immer, A., van der Ouderaa, T., Rätsch, G., Fortuin, V., & van der Wilk, M. (2022). Invariance learning in deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems, 35, 12449-12463. [4] Bondesan, R., & Lamacraft, A. (2019). Learning symmetries of classical integrable systems. arXiv preprint arXiv:1906.04645. [5] Chen, L., Tao, C., Zhang, R., Henao, R., & Duke, L. C. (2018, July). Variational inference and model selection with generalized evidence bounds. In International conference on machine learning (pp. 893-902). PMLR. [6] Graves, A. (2011). Practical variational inference for neural networks. Advances in neural information processing systems, 24.

Questions

1. Can you more clearly separate novelty from background? Right now, a large part of the paper (Section 4) is discussed as if it were a contribution, while it is more of an application of previous derivations. I would also suggest the authors to perhaps write the paper from a more experimental perspective if it turns out the theoretical contributions are not as extensive. 2. Can you elaborate on the proposed evaluation metrics for symmetry comparisons? Moreover, can you explain why there are no other baselines, such as the Laplace approximations or any of the other methods discussed in Section 2.4? Are they not applicable? If so, accentuate why they are not applicable in the text. 3. Can you provide mean and variability metrics in addition to the given values for Tables 1, 2 and 3? Given the lacking novelty and dependence on empirical results, this is a crucial oversight to me. The experiments should be repeated for multiple seeds. 4. Smaller question out of curiosity: it is mentioned that in Section 5.2 (lines 301-302) that the learned Hamiltonian does not generalise to regions further away from the origin due to a lack of data. The symmetries do seem to improve the generalising behaviour, yet are still insufficient, are there any directions for future work to enforce further physical properties to improve out-of-distribution generalisation? If so, an outlook to such a direction could be interesting.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The authors briefly mention that the experiments are rather small scale, but only in the author checklist. It would be appreciated if this is mentioned in the main body of the paper as well. Additionally, the remark on lines 301-302 about generalisation shortcomings could also be highlighted. Finally, the lack of strong guarantees about the quality of the selected/optimised model due to the use of variational inference could also be mentioned, i.e. VI does not guarantee in any way that the variational posterior is in fact invariant or even close to invariance to the learned symmetries. Only experimental evidence can be given in this regard, which luckily shows positive results.

Summary

This paper proposes Noether's Lazor, a Bayesian framework incorporating learnable symmetries for Hamiltonian neural networks. It parametrizes a hidden symmetry as a flow (single-parameter group) derived from the system's conserved quantity. The flow is then applied to the Hamiltonian as it transforms the system's states while conserving the quantity. The objective function is given as the ELBO that averages over the transformation, forcing the Hamiltonian to be invariant, corresponding to the symmetry. Numerical experiments show that the proposed method finds true symmetries.

Strengths

1. This study addresses the challenging but essential problem of finding symmetries in a Hamiltonian dynamical system. 1. Extracted symmetries in the experiment (Fig 5) are interpretable and interesting.

Weaknesses

1. It is not mentioned enough how the proposed method differs from previous studies. For example, [Alet et al., 2021] might be one of the most relevant works but only mentioned that it uses a validation set. Also, it says that `Similar lower bounds to invariant models that average over a symmetry group have recently appeared in prior work [van der Ouderaa and van der Wilk, 2021, Schwöbel et al., 2022, Nabarro et al., 2022]`, but the differences are not mentioned. 1. The computational complexity is not presented. Since the computation of a Bayesian method is usually heavy, it would be nice to mention how it is compared to the vanilla HNN. 1. The concepts of Noether's theorem and Occam's Razor do not sound tightly connected. As Theorem 1 says, Noether's theorem implies that a Hamiltonian H is G_O invariant, where G_O is the flow of an observable O, iff O is conserved under the system of H. It seems the authors use this notation as a prior of H to impose symmetries (symmetrization of H). However, it takes the integral over the transformation parameter \tau (5). This integral is independent of what we do to obtain the marginal likelihood --- the latter takes an average over the parameter of H. So, I think we can use the symmetrization of H without the Bayesian manner, and I cannot find any special reason why we have to use Noether's theorem and Occam's Razor simultaneously. (If I say something wrong, please correct me.)

Questions

1. I think we can think of two baselines: (a) Bayesian but not symmetry-adapted, i.e. the objective is given by E_\theta[ loss of HNN ] + KL, and (b) symmetry-adapted but not Bayesian, i.e. the objective is Eq.(8) but no KL and no expectation w.r.t. \theta. What would their performance be in the experiments? 1. Why did you use Bayes instead of CV? What are the pros/cons? 1. Can you measure the wall clock time of the vanilla HNN and the proposed method? Minor issues: 1. C(x) first appears at line 85 without a formal definition. 1. At line 245, H_{\theta, \eta}( \Phi^{\tau}(x) ) would be H_\theta(\Phi^{\tau}_{\eta}(x)) since the symmetrizing parameter \eta is used in the time evolving operator \Phi but not in H.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations are not addressed.

Summary

This paper proposes a Bayesian framework to learn conserved quantities (and, implicitly, symmetries) in the context of Hamiltonian systems. The idea is that the Hamiltonian system has specific parametric conserved quantity (given by a quadratic function, line 136), in that case the conserved quantity can be learned from data in a Bayesian setting (using the parameterization of the conserved quantity as a prior over the possible Hamiltonian systems) via ELBO. Even though the setting is a bit restricted, the idea is quite intriguing and conceptually very interesting. My main criticism of the paper is that the methodology is not explained in a clear, self-contained way, making the paper hard to follow for the NeurIPS audience. Since the main contribution of the paper is conceptual, I think this is a major drawback. The paper seems hastily written (it has many typos, including in the abstract "conversation laws" instead of "conservation laws", inconsistent notations, and incomplete definitions) and it could significantly benefit from a careful major revision.

Strengths

The topic is very interesting. Incorporating symmetries via conservation laws in machine learning models is an interesting idea in the context of physics informed machine learning, and machine learning models for physical systems. The combination of Noether's theorem with Bayesian modeling is novel. The empirical evaluation, though limited, show that these ideas can be promising.

Weaknesses

In my opinion, the paper would benefit from a major revision focused on improving the exposition to clearly explain the methodology. The paper considers a specific example, where the conservation law is parameterized by a quadratic function. Under this assumption we have an explicit formula for the flow $\Phi$ that seems to be key for implementing the optimization. Without this assumption it is unclear to me that the problem is computationally tractable. I don't see this as a weakness necessarily (though it would make sense to state it as a limitation in the conclusion, if that's the case). However, it does make the main contribution of the paper to be mostly conceptual. Therefore this paper main contribution should be the explanation of the methodology. In my opinion this explanation is not sufficiently clear. In particular it'd be nice to see an explanation of why one would choose to optimize the ELBO over direct optimization over the parameters that define the conservation laws and the Hamiltonian. **Comments:** - Line 85: The definition of the bracket depends on C which has not been defined before. Should it be O? - Please make explicit what is the domain and range of the functions considered in the manuscript. For example $x: \mathbb R \to \mathcal M$ - This sentence could be explained better: "We have used a different symbol to not conflate the ODE time $\tau$ with regular time $t$ of the trajectory generated by the Hamiltonian."? In particular in relation to referring $\tau$ as "symmetry time". - Also, the difference on the parameters $\eta$ that define the space of symmetries vs $\theta$ that define the space of Hamiltonians should be made more explicit (section 3.1). In particular, this should be explained in section 4. - Section "Why the marginal likelihood can learn symmetry". What does it mean to "learn symmetry" mathematically in this context? Does it mean to learn $\eta$? - A few typos: "conversation laws", "train data", "generalisaiton", "als".

Questions

Is an explicit formula for $\Phi$ needed in order to implement the optimization? Is that a limitation for the approach? What is the relevance of the distribution over $\tau$? How do the results depend on it? Can we used this to learn the specific conservation law ($\eta$) or is it only known implicitly in the solution $H(\Phi)$?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The methodological limitations are not explicitly stated.

Summary

The authors propose to use the parameterized symmetries for learning correct Hamiltonian dynamics from data. It is based on the Noether’s theorem, which states that the continuous symmetries generated by observables O for Hamiltonian H yields the conservation of O, and vice versa. To do this, the authors parameterizes the observables as a quadratic form, and makes the learnable Hamiltonian function (neural network) to be invariance with respect to the one-parameter group (flow) generated by O, which is analytically expressible. Invariance is achieved through techniques such as orbit pooling, which averages the outputs of the Hamiltonian over the group orbit. To learn the Hamiltonian with proper symmetries, the authors propose to use the Bayesian approach, i.e., learning the ELBO, thus the title of the paper is the “Noether’s razor” (similar to the Occam’s razor that regularize the model with proper weights). The authors validate their approach to classical examples like Harmonic oscillator and n-body systems, which showcase the method can model the appropriate dynamics as well as find a correct symmetries.

Strengths

The motivation of this paper is clear, and highly relevant to both the AI + Science and geometric machine learning communities. The paper is generally well-written and easy to follow. The method is based on well-known Noether’s theorem, thus is very principle. Overall, I believe this paper is worthy of publication and will provide valuable insights to researchers in the relevant field, even though it may not be particularly striking to a broader ML audience.

Weaknesses

- This paper assumes that the symmetry arises from the symplectic flow of a quadratic form, which is a significant assumption that lacks a clear explanation. The authors should justify the use of a quadratic form, demonstrating that it is a reasonable assumption for the systems of greatest interest (not only for the benchmarks used in the paper). - The experiment consists only of low-dimensional examples modeling phase space trajectories directly, raising doubts about whether the proposed results can be applied to high-dimensional data. Please note that even the original Hamiltonian neural network [1] includes a task involving learning the underlying dynamics from images.

Questions

The authors directly parameterize the model Hamiltonian to be invariant by averaging over the group orbit. Another way to achieve the symmetry is satisfying {C, H} = 0, according to the Noether’s theorem. Since the latter can be achieved easily by adding the regularizer like ||{C, H}||, I am curious about why the authors chose to use the method proposed in the paper instead of this regularizer, and how much of a performance difference there is between the two methods. Some typos: - page 2, line 70, $x_t + J \nabla H_{\theta} (x_t) \Delta t$ - page 3, line 85, $\nabla O^T(x) \cdot J \nabla H (x) $ - page 3, line 120, also - page 4, line 156, the the

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do not explicitly mention the limitation in the main manuscript. Please refer and address Weaknesses.

Summary

This work aims to model symmetries for strong inductive biases in machine learning models of dynamic systems. Instead of constraining models to certain symmetries, this work focuses on automatically learning them from data. It proposes to parameterize symmetries using conserved quantities by Noether's theorem. The conserved quantities are further incorporated into priors and the model is trained by optimizing a lower bound of the marginal likelihood, which is able to balance both data fit and model complexity. Empirical results on both n-harmonic oscillators and n-body system problems are presented.

Strengths

- The integration of Noether’s theorem with variational inference to learn conserved quantities for symmetries is a novel and significant contribution, providing a new perspective on symmetry learning. - Experiments show that the learned symmetries have the same rank as the ground truth symmetries, showing the effectiveness of the proposed learning method. Further from the empirical results, the model with learned symmetries has a similar performance to the one with built-in oracle symmetries and outperforms the ones without. - The work is overall well-written and has provided sufficient background for the authors to understand the proposed method as well as references to existing work.

Weaknesses

- This work can be further improved by providing computational analysis on the computation overhead induced by symmetry learning. Maybe the authors can elaborate on this point. - From Section 5.1, it seems that the choice of hyperparameter K has to be greater than the dimension of ground-truth symmetry in order to capture the full symmetries. I wonder how to choose such hyperparameters in practice.

Questions

- See weakness.

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Overall rebuttal

We thank all reviewers for the feedback and help to improve the paper. We are excited to have received a "9: Very Strong Accept", and are confident that the concerns raised by the lower score reviews are sufficiently addressed in this rebuttal. Overall, most reviewers found the paper very 'clearly written', 'principled', 'easy-to-follow' and 'highly relevant to both the AI + Science and geometric machine learning communities'. In particular, we are proud that reviewers found the paper 'intriguing and conceptually interesting’, and were positive about the empirical validation and deemed the results promising. Most raised concerns are addressed in direct rebuttals. Since several reviewers raised questions about the use of the quadratic form, we also provide a more thorough discussion on this issue in this overall rebuttal: > Quadratic form We indeed consider learnable conserved quantities of the quadratic form. Regardless of the quadratic form, our model can in principle still learn any dynamical system, also those with non-quadratic symmetries, given a sufficiently flexible network. This is already a direct improvement over models without learnable invariance properties, such as regular HNNs, while also capturing commonly encountered symmetries that are currently hard-wired into architectures. Regarding the quadratic form as a type of learnable symmetry, we feel that this is not an overly strong assumption. Quadratic conserved quantities can represent any symmetry that has an affine (linear+translation) action on the state space. This includes essentially all cases that are currently studied in geometric deep learning. Note that the quadratic constraint does not limit the shapes of the symmetry groups that we learn. As an example, in our experiments we find the Euclidean group, which is itself a curved manifold with non-trivial topology. Hence, we find this quadratic assumption to be not overly strong. However, we welcome suggestions for common examples in deep learning with equi/invariance to groups with a non-affine action, so that we can list these examples as limitations of our method. The technical reason why quadratic conserved quantities can represent complex groups, is that the learned conserved quantities are a basis of the Lie algebra of the Lie group of symmetries, with the Poisson bracket being identical to the Lie bracket of the Lie algebra. The structure of the Poisson bracket between the conserved quantities determines the shape of the group, and this is not constrained by the conserved quantities being quadratic. If one wishes, we can extend our framework to non-quadratic conserved quantities to model non-affine actions. This has the downside that the symmetrization needs to be done with an ODE-solving, as in Eq. (3), instead of with the matrix exponential. This slows down the implementation by a large amount. Furthermore, such free-form conserved quantities have many more parameters, which complicates the Bayesian model selection. We did implement this and will include it in the code that will be released upon acceptance. We have omitted the details on free-form conserved quantities from the present manuscript for space reasons, but will elaborate on it in the appendix of the final version.

Acknowledgement of author rebuttal

Thank you for your thoughtful answers to my concerns, please consider my follow-up below. **Novelty.** Thank you for further differentiating your work from previous work. I do now better see how using Noether's theorem allows the proposed method to model more complex symmetry groups by modelling their conserved quantities. This does definitely increase the impact and I also agree with the authors that quadratic forms might look simple, though they can lead to the modelling of very non-trivial symmetry groups. I would urge the authors to heavily emphasise this contribution in the main paper, stressing that previous work only modelled simpler affine groups. With respect to the effort of scaling VI for model selection, could you please specify precisely what you contributed here? From appendix D I can see that you started from the work [1] and found an improved closed expression for a special (?) case where the KL is computed between a non-diagonal Gaussian and a zero mean scalar variance Gaussian. This could be an interesting contribution, but it is hard to gauge why this improvement was necessary or why this special case was chosen. What where the bottlenecks that led you to this result and why do you need it? Additionally, since you argue it is crucial to scale VI for model selection, why is it not given more attention in the main text? Especially considering structured VI is an active area of research [2, 3]. Lastly, I do maintain that much of the general methodology of the paper is extremely close to previous work, such as [4]. Specifically, Section 3.2 discusses how to approximate the integral in Equation 5 (Equation 1 in [4]) in the same way as [4] did (Sections 3.4 and 3.5 in [4]). Moreover, the contents of Section 4.2 and 4.3 are also similar to Sections 3.6 and 3.7 of, for example, [4]. For the latter there is a reference, but it is not disclosed how different or similar the expressions are. Using the same methodology is completely fine, but the lack of proper references makes the contents look more novel than they are. Please amend this and properly cite the relevant literature. **Metrics.** While not super high-dimensional, 100 dimensions is also not a small number. I would, for example, be curious how using the cosine similarity (distance) influences the results. Considering the novelty of the paper is now more clear to me, I will increase my score to "5: borderline accept". I am willing to increase my score further if the authors can additionally clearly indicate that part of the methodology has been used in previous works. More clarification on the possible contributions to scaling VI might also positively influence my score. [1] Louizos, C. and Welling, M., 2016, June. Structured and efficient variational deep learning with matrix gaussian posteriors. In International conference on machine learning (pp. 1708-1716). PMLR. [2] Hoffman, M.D. and Blei, D.M., 2015. Structured stochastic variational inference. In Artificial Intelligence and Statistics (pp. 361-369). [3] Lindinger, J., Reeb, D., Lippert, C. and Rakitsch, B., 2020. Beyond the mean-field: Structured deep Gaussian processes improve the predictive uncertainties. Advances in Neural Information Processing Systems, 33, pp.8498-8509. [4] van der Ouderaa, T.F. and van der Wilk, M., 2022, August. Learning invariant weights in neural networks. In Uncertainty in Artificial Intelligence (pp. 1992-2001). PMLR.

To authors

I appreciate that the authors carefully responded to my review. My concerns regarding prior studies, Q1, and Q2 are now resolved. I hope the authors will address the issue of computational cost. Although I'm not fully convinced about the design choice to combine Noether's theorem and Occam's razor, I admit it works in some scenarios. I will raise my score.