Symmetry-Informed Governing Equation Discovery

Summary

The authors developed an approach that allows exploitation of symmetry in common equation discovery algorithms and test their approach on dynamic systems with and without symmetry. For this they make use of Lie groups.

Strengths

- Despite Lie groups being a new topic for me, I was able to follow the well written high-level explanations in Sections 3.1 and 3.2 - I like the approach that was used to exploit additional structure in the search - The results are very strong on the chosen systems, though I am still left sceptic of _how much_ the results are better in Table 1. - The experiments section is written concisely.

Weaknesses

_Weaknesses_ - Just looking at Figure 1, I find it hard to recognize at which points the dynamics are searched, i.e. where the equation discovery itself is happening. From the text prior to Figure 1, I expect the authors approach to find the symmetry first, constrain the search space and then perform equation discovery, but Figure 1 suggests otherwise. - Chapter 4 could benefit from examples (or a running example) how corresponding functions might look like, to facilitate understanding. In particular since $M_\Theta$ can be computed in a symbolic fashion. Since the work is already tightly packed, such examples can also be mentioned in the appendix. _Minor complaints/typos_ - A small arrow/text indicating the start "Start" in the flowchart on the left side of Figure 1 would make orientation slightly easier. - Section 4 was harder to read, compared to section 3 - 261: With RK4 you mean Runge-Kutta 4 methods? - Figure 2 could be bigger to improve readability. To summarize: I enjoyed this work and find it relevant to the community and a good contribution. However, the explanations in Chapter 4 were partly unclear to me and could benefit from an additional revision. The most conflicting point for me are the results of Table 2, which I hope to have clarified during the discussion.

Questions

I want to highlight a major question before all others. With respect to the results in Table 2: - If there is no symmetry in the system, shouldn't your approaches fallback to their respective base approaches, i.e. SINDY and GP? Why does your approach still outperform them so consistently, despite there being no exploitable symmetry in the ODEs? - Did you look into the results of the LaLiGAN? Did you recover the non-existence of symmetries and _still_ performed better? Or did it find some symmetry which your approach then used? Other questions: - 78/79: Thank you for highlighting works that already exploit symmetries. Can you provide a brief description of how you differ from Loiseau & Brunton 2018 and Guan et al. 2021? - I have a minor question for Prop 3.2. What does it mean for the flow map $f_\tau$ to be equivariant to the G-action on X? Does that mean that $f_\tau$ needs to still be a valid flow map after an element of the symmetry group has been applied to $X$? - For Theorem 3.3 I have a few minor questions: - What is $\epsilon$ in the $g = \exp(\epsilon v)$ term? - Why is $g$ defined this way? Would it not be possible to calculate the Jacobian otherwise? - Is it guaranteed that $g \in G$? Does that only hold because $G$ is a symmetry group? Or generally? - I assume that $v$ is a matrix, please correct me if I am wrong, what does it mean to perform $\exp(\epsilon v)$ in that case? Is this the matrix exponential? - What is $L_v$ in Proposition 4.1? Is it a linear transformation? - 164/165: I don't understand what $c$ is and what its purpose is - 180: The set of polynomials doesn't include products $x_1 x_2$ like in the example for SINDY in line 163, correct? - 183: Is $W$ vectorized with respect to rows $\{[w_{11} ... w_{1n}], ...\}$ or columns $\{[w_{11} ... w_{n1}], ...\}$? - Where does the factorization in Equation (10) come from? Are there established methods that easily generate this? What do its components mean?

Rating

7

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

In my opinion the only limitation of this work lies in the experiments in 5.3, as I stated before. I hope the authors can either point out how their approach differs from vanilla SINDy/GP and explain why it is better or demonstrate, perhaps using a "neutral" operator instead of LaLiGAN, that it does fall back and that the difference is due to LaLiGAN.

Summary

This paper propose to leverage the symmetry to discover underlying dynamics (especially the ones described by autonomous ODEs) correctly from data. Specifically, the proposed method combines conventional symbolic regression approaches like SINDy with the symmetry-based constraints or regularizations. The used symmetry-based regularization is derived from the Lie group symmetry, namely the flow of the given dynamics should be $G$-equivariant with respect to the given Lie group $G$ of symmetries. For the case that the group of symmetry transformations is unknown, the authors use some recent techniques that can learn the unknown symmetry from data, and parameterize the Lie algebra of the learned symmetry with neural networks. In this case, the extracted symmetry regularization can be encoded by using the infinitesimal version of $G$-equivariance with the learned Lie algebra generators. The authors validate their proposed approach with some dynamical systems.

Strengths

**Significance.** Using symmetries to generalize the machine learning dynamics is very relevant topic. The combination of the symbolic regression with the symmetry regularization for recovering the underlying dynamics is thus very principle. *** **Clarity.** The paper is well-written and easy to follow. The main paper is concise and highlights the advantages of the proposed technique without heavy math/details. Readers who want to know the detailed background and techniques on the proposed method can find the details in the appendix.

Weaknesses

**Novelty.** The proposed technique is straightforward, if one already knows the group of symmetry transformations that should be applied. It is basically the regularization for the (infinitesimal) $G$-equivariance of the given Lie group $G$, i.e., $\lVert f(g \* x) - g \* f(x) \rVert$. Therefore, the interesting part of the proposed method is the case that the symmetry is unknown. In this case, the authors rely on [1], which can find the infinitesimal generators of Lie group of symmetries from data. Because they use [1] without clear modification/enhancements, there is some doubt as to whether the combination of two already known techniques (equivariance regularization and symmetry discovery) can be sufficiently novel. [1] Yang et al. Latent Space Symmetry Discovery. arXiv 2023. *** **Experiments.** The proposed method is evaluated on relatively simple dynamics. It would be beneficial if the authors could provide experimental results on more complicated dynamics, such as chaotic systems. *** There are some typos, e.g., the denominator of (35), and the summation spaces ($v$ instead of $g$) of (35) and (36)

Questions

The author mentions that though there are four forms of Lie point symmetry (i.e., equations (4 – 7) or equations (34 – 17)), they are empirically similar in terms of the performance. What would happen if the authors use multiple ones simultaneously instead of just one of these four? For example, how would the performance be if you use both (34) and (35) at the same time? I think this would work similarly to the Sobolev training [2], thus potentially improving performance, especially when the dataset is noisy. [2] Czarnecki et al. Sobolev Training for Neural Networks. NeurIPS 2017.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors mention that their method only consider the time-independent point symmeties for autonomous ODEs. They provides some potential future research directions for generalizing their approaches, e.g., for non-autonomous ODEs.

Summary

The authors consider an estimation technique similar to the well known SINDy technique for recovering interpretable parameterizations of ordinary differential equations (ODEs). The authors primarily consider the context where there exists a Lie symmetry that constrains the solution space of ODEs to search over. The authors describe different ways in which to incorporate the symmetry information within the learning problem. The first is to consider linear constraints in which case the problem simplifies into another coefficient regression optimization. The second involves general constraints for which the authors propose a regularization scheme to constrain the function class. These methods are described in terms of both the case of observed data and a learned coordinate system. Finally, the authors consider a few different experiments where they try to estimate the coefficients for known data as well as predict the evolution of the data forward in time.

Strengths

Including the symmetry information is a natural approach to try to constrain the learning problem into something more manageable. The general SINDy learning problem can be difficult to interpret due to the requirement of the specification of the basis. In that sense, this method provides a good solution for imposing some constraints on the function space to make the problem more tractable. The numerical results suggest the method performs very well compared to the baselines. For the forward prediction on the reaction diffusion system, the method also performs better than the related baselines. This shows the promise in some more general tasks such as time series forecasting given sufficient regularity on the observed data.

Weaknesses

The method largely focuses on cases where symmetries are linear. The regularization approach in equation 11 seems a bit arbitrary in its formulation (e.g. using the relative error) as well as it seems like the results are primarily applicable in the cases where the symmetry is linear. Some of the empirical results are slightly confusing, in particular when the baselines fail catastrophically under some circumstances leading to high RMSE. However, in the cases where the the baselines successfully identify the equation, the baselines tend to have smaller RMSE. This may be a hyperparameter tuning issue, but it would seem more intuitive that the method with the symmetry should be easier to achieve a lower RMSE.

Questions

Is there any existing theory on how well learned the symmetries can be from the data? How does the computational cost of the method compare? I'm particularly interested in the increase in cost associated with the vector Jacobian product computation. Is there any intuition why the RMSE of W-SINDy is better than the proposed method in the case of successful identification?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors briefly discuss limitations regarding requiring known symmetries for the algorithm. It should probably be discussed more on how performance changes when using prior knowledge versus when learning these symmetries from the data.

Summary

The paper proposes to leverage symmetry to guide the equation discovery process, compress the equation search space, and improve the accuracy and simplicity of the learned equations. Depending on the types of symmetries, the paper develops a pipeline for incorporating symmetry constraints into various equation discovery algorithms, including sparse regression and genetic programming

Strengths

1. The paper clearly establishes a pipeline to use Lie point symmetries of ODEs to constrain the equation learning problem. 2. The paper theoretically derives the criterion for symmetry of time-independent ODEs in terms of equivariance of the associated flow map. 3. From the above mentioned criterion, the paper solves the constraint explicitly to compress the equation search space in sparse regression and promote parsimony in the learned equations. 4. In experiments across many dynamical systems with substantial noise, the symmetry-informed approach achieves higher success rates in recovering the governing equations.

Weaknesses

The Time-Reversal Symmetric Ordinary Differential Equation Network has been proposed [1]. However, this work does not cite the relevant article [1]. While the presentation may differ, the symmetries investigated in both studies are fundamentally the same. Although this work is based on SINDy and the relevant article [1] is based on Neural ODE, this may not be an essential difference. Therefore, the reviewer recommends a detailed comparison of the two approaches, highlighting their similarities and differences. Additionally, incorporating comparisons into the experimental section would be beneficial. [1] Huh, I., Yang, E., Hwang, S. J., & Shin, J. (2020). Time-reversal symmetric ode network. Advances in Neural Information Processing Systems, 33, 19016-19027.

Questions

1. Why set 5% and 20% noise respectively in the experiment in Section 5.1? What happens when the error scale is smaller? 2. For equation discovery with unknown symmetry, if we assume that the symmetry is unknown in the first two examples, what will be the result? 3. The paper only considers two-dimensional situations. What are the results for high-dimensional examples?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper focuses on incorporating symmetries to equation discovery pipelines for ODEs. If the governing equation has a known linear symmetry (its solutions are invariant with respect to a known linear action by a known Lie group), this paper derives a set of conditions the ODE needs to satisfy, and incorporates those conditions into a symbolic regression framework such as SINDy. When the symmetry is still known but not linear, the paper derives a form of regularization to promote that symmetry into the learning process. This procedure wasn't completely clear to me. When the symmetry is not known, the paper uses LaLiGAN, a symmetry discovery framework, to learn the corresponding regularization.

Strengths

This is a well-written paper in an interesting topic: imposing symmetries in governing equation discovery. It considers approaches from the simpler case (known linear symmetries) to the more general case (unknown non-linear symmetries), and it provides numerical evaluations for them.

Weaknesses

The methodology behind the regularization approach and the LaLiGAN approach were not clearly explained in the paper. It would be good if the authors can provide a more detailed explanation. In particular the technical approach for unknown non-linear symmetries (based on LaLiGAN) was not clear to me. I went to the recent LaLiGAN paper to see how it works and I found that the original LaLiGAN paper already proposes the approach for governing equation discovery. I suggest the authors explain how the approach they propose in this paper differs from the approach proposed in LaLiGAN. Moreover, the LaLiGAN paper does not have a github repository with their code, so the authors should be able to make sure that all their code and their dependencies can be made publicly available if the paper gets accepted.

Questions

What are the limitations of the approach in the known non-linear symmetries case? What are the limitation of the approach in the unknown symmetries case?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors could comment on the technical limitations of their work (e.g. dimensionality, ODE order, sample complexity, etc).