Many successful methods to learn dynamical systems from data have recently been introduced. However, ensuring that the inferred dynamics preserve known constraints, such as conservation laws or restrictions on the allowed system states, remains challenging. We propose stabilized neural differential equations (SNDEs), a method to enforce arbitrary manifold constraints for neural differential equations. Our approach is based on a stabilization term that, when added to the original dynamics, renders the constraint manifold provably asymptotically stable. Due to its simplicity, our method is compatible with all common neural differential equation (NDE) models and broadly applicable. In extensive empirical evaluations, we demonstrate that SNDEs outperform existing methods while broadening the types of constraints that can be incorporated into NDE training.
Paper
References (77)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
Neural ODEs which learn a particularly dynamics can be unstable during evolution of the learned dynamics. This papers studies instabilities which take solutions outside of the constraint space of a given evolution. The authors show that a simple added control term to the learned evolution can improve stability noticeably. This added term, intuitively, projects part of the flow onto the tangent space of the constraint manifold. This stability term has the benefit that it is not complicated to implement, has some theoretical guarantees of stability, and works well in the models they consider.
Strengths
As mentioned in the summary, I see a few benefits to the authors' proposed techniques: - The method is rather simple and easy to follow. The reason why it enhances stability is readily apparent and experiments back this up. - The authors raise an important point about neural ODEs in the context of constraints. As someone not completely familiar with the field, I appreciated how they setup the problem and discussed it. - There are some simple theoretical guarantees of the stability of the approach. The experiments back up these theoretical guarantees, though the experiments appear to be focused on mostly toy models.
Weaknesses
Main comments: - There are a host of tools in the Riemannian optimization literature that are relevant here and not discussed. For example, there exists many algorithms for learning flows or vector fields on manifolds (e.g. [1], [2]) and performing transport on those manifolds. There are also simpler methods such as projected gradient descent or riemannian descent via retractions of the exponential map which can easily stay on a manifold and could be adapted to this setting. I suppose the authors would argue that their idea is simpler, but I would not be convinced unless these are compared to at least empirically or theoretically. - The authors' main contribution is addressing a control problem and not really a learning problem. The paper includes an additional term to the ODE evolution to better control the trajectory, but this parameter is not learned in any way. Thus, the problem they address is really one of control and not learning. This problem of stable control is well studied in control theory as far as I understand. I raise this point for two reasons. First, I worry the problem the paper addresses is a bit outside of the scope of this conference. Second, as I am not an expert here, it is not clear to me how effective their particular controller is. Perhaps the authors can expand on this point and address comparisons to methods in the control literature. - The theoretical guarantees and experiments, I think, could be expanded. Related to my previous comments, the theorems are two straightforward statements about the added stability of the control term. There are no theorems relating to how much better this is for specifically learning tasks (e.g., some proofs about generalization error). On the experiments side, all experiments seem to be on toy models of small dimensional systems. It is difficult to see how much better this will work in higher dimensional or more complex systems. Smaller comments: - Eq. 3 missing $u \in \mathbb{R}^n$ - A number of references have no journal or indication of where it was published (even an arXiv indicator) - The pseudo-inverse can be expensive to calculate in high dimensions with large numbers of constraints. Is this an issue in practice? Have you considered faster approximations to this? - Some portions of the text use $f_\theta(u)$ and others $f_\theta(u,t)$. I would stick with one notation and express whether or not the setting the authors study depends on $t$ explicitly throughout the work. - All of the experiments seem to be on low dimensional systems. Is there a reason why the authors did not go to higher dimensional examples? - Many of the problems people care about are on continuous spaces (e.g., fluid flow in Navier stokes). Does the method here extend to that setting in an obvious way? References: [1] Lou, Aaron, et al. "Neural manifold ordinary differential equations." Advances in Neural Information Processing Systems33 (2020): 17548-17558. [2] Ben-Hamu, Heli, et al. "Matching normalizing flows and probability paths on manifolds." arXiv preprint arXiv:2207.04711 (2022).
Questions
Please see weaknesses section for questions.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
There is a brief discussion of limitations of the work in the conclusion section.
Response to "smaller comments"
Due to the character limit, we respond here to the reviewer's "smaller comments". > - Eq. 3 missing $\in \mathbb{R}^n$ > - A number of references have no journal or indication of where it was published (even an arXiv indicator) We thank the reviewer for bringing these shortcomings to our attention and we have fixed them in the revised manuscript. > The pseudo-inverse can be expensive to calculate in high dimensions with large numbers of constraints. Is this an issue in practice? Have you considered faster approximations to this? This has not been an issue for us in practice and we have not yet empirically investigated faster approximations to this or alternative choices for the stabilization matrix $F$. However, as also outlined in our respond to reviewer **pbb8**, the formulation is quite general and other options are certainly possible. For example, Chin [4] reports competitive results by simply using the transpose of the Jacobian, which may be attractive when the pseudoinverse becomes expensive, e.g. when there are many constraints. An alternative possibility for when the pseudoinverse becomes expensive is to train initially with no stabilization, adding stabilization only towards the end. Finally, we highlight again that the complexity of the pseudoinverse is $\mathcal{O}(m^2n)$, where $m$ is the number of constraints and $n$ is the dimension of the problem, highlighting that our approach scales well to high dimensions. > All of the experiments seem to be on low dimensional systems. Is there a reason why the authors did not go to higher dimensional examples? The five examples in our paper were carefully chosen to test our method across a wide range of settings, greatly expanding the types of constraints that can be incorporated into neural ODE training: - First- and second-order systems - Hamiltonian and non-Hamiltonian systems - Autonomous and non-autonomous systems - Chaotic and non-chaotic systems - Purely data-driven and hybrid systems - Time-dependent and time-independent constraint manifolds > Many of the problems people care about are on continuous spaces (e.g., fluid flow in Navier stokes). Does the method here extend to that setting in an obvious way? As we also outlined in our response to **gVTB**, our method is also applicable to discretized PDEs when the constraints can be formulated in terms of the discretized variables. In this context, we again emphasize that our method scales linearly in the number of dimensions, implying that our method should remain competitive when extended to PDEs. However, while we actively work on the PDE case, we do consider a rigorous treatment of PDEs to be beyond the scope of the present paper and would like to outline it as a subject for a follow-up study. [4] Chin, H. S. (1995). Stabilization methods for simulations of constrained multibody dynamics (T). University of British Columbia. Retrieved from https://open.library.ubc.ca/collections/ubctheses/831/items/1.0080031
Some follow-up
Dear authors, Thank you for the write-up and responding to my questions and criticisms. I think the authors are missing something in my main critique, and I would like them to further clarify. I pointed out the two works by Lou et al. and Ben-Hamu et al. as examples of possible methods that can be used to explicitly deal with optimization on manifolds. These are not meant to be the only two works as there is a whole field of work surrounding how to enforce constraints in neural networks and/or performing optimization on manifolds. Let me give some more concrete points: - There are a number of packages and tools for Riemannian optimization that need little modification to use [1-4]. Indeed, using charts is one approach, but there are other ways to explicitly encode the manifold in these works. These methods often work out of the box. - Even outside of Riemannian optimization, there are works that by design ensure neural networks satisfy conservation equations [6]. - As yet another example, for the experiments constrained on the sphere, there are a number of works [7-8] that have done extensive work on efficiently performing optimization on the unitary/orthogonal manifold especially for neural networks. - There are also a number of methods to handle optimization with constraints in differentiable ways [9-10]. I do not see comparisons to any of the above in the work you are doing, and it makes me hesitant to say that your algorithm works better than any existing ones. At the very least, a comparison needs to be made to these methods. I am willing to consider the point that your algorithm is more general but that does not exclude comparisons to these (and I'm still not convinced on the generality either). Can the authors address this point in more detail? **References:** [1] Boumal, Nicolas, et al. "Manopt, a Matlab toolbox for optimization on manifolds." The Journal of Machine Learning Research 15.1 (2014): 1455-1459. [2] Boumal, Nicolas. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. [3] Xiao, Nachuan, et al. "CDOpt: A Python Package for a Class of Riemannian Optimization." arXiv preprint arXiv:2212.02698 (2022). [4] Smirnov, Oleg. "TensorFlow RiemOpt: a library for optimization on Riemannian manifolds." arXiv preprint arXiv:2105.13921 (2021). [5] Baddoo, Peter J., et al. "Physics-informed dynamic mode decomposition." Proceedings of the Royal Society A 479.2271 (2023): 20220576. [6] Richter-Powell, Jack, Yaron Lipman, and Ricky TQ Chen. "Neural conservation laws: A divergence-free perspective." Advances in Neural Information Processing Systems 35 (2022): 38075-38088. [7] Lezcano-Casado, Mario, and David Martınez-Rubio. "Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group." International Conference on Machine Learning. PMLR, 2019. [8] Kiani, Bobak, et al. "projUNN: Efficient method for training deep networks with unitary matrices." Advances in Neural Information Processing Systems 35 (2022): 14448-14463. [9] Donti, Priya L., David Rolnick, and J. Zico Kolter. "DC3: A learning method for optimization with hard constraints." arXiv preprint arXiv:2104.12225 (2021). [10] Amos, Brandon, and J. Zico Kolter. "Optnet: Differentiable optimization as a layer in neural networks." International Conference on Machine Learning. PMLR, 2017.
Response (1/3)
We thank the reviewer for further clarifying their main critique. We start with our two main arguments, which we will subsequently elaborate on. We will add these clarifications to the related work part of the final manuscript. 1. We agree that there are relevant connections between neural ODEs on Riemannian manifolds (as we study them) and the vast literature on optimization on Riemannian manifolds (aka Riemannian optimization). However, **we disagree that this literature holds ready-to-use baselines that we can compare to**. 2. Again, we fully acknowledge that various communities have studied optimization problems on certain (types of) manifolds. However, **we argue that the requests for comparisons reflect an overly broad view of what is “comparable” to our method**, resulting in an infeasible task that would require substantial algorithmic adaptation and innovation and be entirely out of scope. A major contribution of our work is precisely its broad applicability without any case-by-case adaptation needed for specific manifolds. Before supporting these arguments in detail, we want to make sure we are on the same page about the term “manifold optimization”. **Manifold Optimization and Manifold Neural ODEs** “Manifold optimization” aims at finding parameters that optimize some objective function while constraining those parameters to remain on a pre-specified manifold in parameter space. The aim of manifold neural ODEs, on the other hand, is to constrain numerically integrated neural ODE trajectories to remain on a pre-specified Riemannian manifold in state space. In our problem setting, the manifold is defined by explicit constraints on the state variables, and is not directly related to the neural network parameters that are being optimized. We highlight that for manifold neural ODEs it also does not suffice to constrain the trained neural network outputs to the constraint manifold, since NDEs take states as inputs but return velocities, i.e., gradients. The output space of the NDE and the state space in which the constraint manifold lives are not equivalent. **Adapting Existing Methods to our Problem Setting** All of the literature we have surveyed deals exclusively with model Riemannian manifolds that possess convenient closed form expressions for geometric quantities such as exponential maps and their inverses. For our specific setting, where the constraints are given in the form $g(x) = 0$, such closed form geometric quantities are acknowledged in the literature to be difficult to derive in general (see below). As an alternative, the reviewer suggests that existing methods from the Riemannian optimization literature may be adapted “out of the box”. However, as we have outlined, our setting differs from Riemannian optimization in key respects and neither the cited packages (Manopt [1], CDOpt [3], TensorFlow RiemOpt [4]), nor any related packages that we’re aware of, can in fact be applied directly to neural ODEs. Furthermore, Boumal [2] briefly considers $g(x) = 0$ constraints and directly notes with regards to Riemannian optimization that the retraction "may be difficult to compute in general", highlighting that this is a difficult problem even in the context of Riemannian optimization and further limiting the adaptability of these methods to our setting. However, we do not rule out the possibility of further adapting and developing these methods, and we consider it a promising opportunity for further work. At the same time, *they cannot yet be viewed as baselines or competitors for our method*. Finally, for completeness let us also add a brief discussion of existing methods for manifold neural ODEs and how they differ from our approach. **Existing Methods for Manifold Neural ODEs** Many recent works have proposed methods for constructing normalizing flows (NFs) on Riemannian manifolds. While methods for discrete NFs have limited applicability to our setting, since they do not require solving a neural ODE on a manifold, it is nonetheless instructive to consider how they deal with the manifold. Gemici et al. [11] construct NFs on spheres, tori, and their products with Euclidean space by projecting to a homeomorphic Euclidean space, transforming the density using a normalizing flow, and then projecting back. Similarly, Bose et al. [12] construct diffeomorphisms to hyperbolic space, while Rezende et al. [13] construct diffeomorphisms to tori and spheres. This general approach, of hand-crafting transformations on a case-by-case basis, does not extend to our setting of arbitrary constraints. Moreover, even for the special settings we consider where one or the other manifold has been dealt with in the literature, these works all seek to solve a different task and would need to be adjusted for learning ODE dynamics.
Response (2/3)
More interesting in the context of our work are continuous normalizing flows (CNFs) on manifolds, since they typically require solving a neural ODE on a manifold. As we discussed previously, Lou et al. [14] construct a global diffeomorphism by using a local chart to map between the manifold and its tangent space via the exponential map. However, we reiterate that constructing the exponential map on a case-by-case basis for manifolds arising from arbitrary constraints such as ours is not trivial. Ben-Hamu et al. [15] introduce a novel method for training continuous normalizing flows that avoids the need to solve the neural ODE entirely, instead dealing directly with the vector field defined by the neural ODE. They constrain the vector field to the tangent space using projection operators for spheres and product manifolds. Let us now come to our argument 2. Certain manifolds are of particular interest for various tasks such as - spheres, e.g., for modelling phenomena on the globe - Lie algebras such as the orthogongal or unitary group for physics, vision, and robotics applications (including the vast literature on group in/equivariant learning) - hyperbolic spaces, e.g., for modelling hierarchical data or knowledge graphs - and many more. Similarly, various communities have considered learning in settings where some quantity is preserved, such as [6] mentioned by the reviewer. Again, these works span various domains, the most relevant for our work being the works on learning Lagrangian and Hamiltonian systems, which we survey in our related work section. While such conservation laws can (amongst other ways) often be specified as manifold constraints, they form narrow special cases of the constraints our method can handle. Even more importantly, we highlight again that, outside the methods discussed in our related work section, most of these works set out to solve tasks different than learning ODE dynamics and would require substantial adaptations to our task, even in the settings where they are applicable. After these general arguments, we now also reply to the remaining questions/comments directly. We emphasize that this case-by-case differentiation from the cited papers only exemplifies our broader arguments above and that also other works within those fields do not apply as baselines to our setup. >Let me give some more concrete points: There are a number of packages and tools for Riemannian optimization that need little modification to use [1-4]. Indeed, using charts is one approach, but there are other ways to explicitly encode the manifold in these works. These methods often work out of the box. As outlined in more detail above in support of our argument 1, these tools, specifically designed for Riemannian optimization, cannot be used out of the box in the context of neural ODEs on manifolds. Adaptations to our setting are not obvious and require substantial novel contributions in their own right. > Even outside of Riemannian optimization, there are works that by design ensure neural networks satisfy conservation equations [6]. The conservation laws studied in this paper are based on the continuity equation, for example, conservation of mass in fluid flow, and are not applicable to the present ODE setting (see also our support of argument 2). > As yet another example, for the experiments constrained on the sphere, there are a number of works [7-8] that have done extensive work on efficiently performing optimization on the unitary/orthogonal manifold especially for neural networks. These works, and parameterizations of other Lie groups or manifolds more broadly, cannot be viewed as baselines for our method. In our setting, it is not the neural network output that is constrained to a manifold. Hence methods that parameterize (some specific) manifolds via neural networks are not directly applicable to learning ODE dynamics where the state space is constrained. Following our argument 2, even if they were, these existing methods are only capable of parameterizing certain specific manifolds and lack the generality of our method.
Response (3/3)
> There are also a number of methods to handle optimization with constraints in differentiable ways [9-10]. DC3 (like related methods to incorporate hard constraints) concerns the typical constrained optimization setting, where we seek to find a single set of parameters that optimize the objective while satisfying the constraints. In our setting, we do not constrain the final parameters of the learned neural network (in fact the constraint manifold is not in parameter space), but the entire trajectory when integrating the learned network needs to stay entirely on a manifold (living in the state space). There is no direct way of extending DC3 or related methods that exploit neural networks for constrained optimization. In OptNet, the authors develop layers that themselves solve a quadratic program, i.e., perform a “simple” optimizaton by themselves. These QPs can also contain linear inequality and equality constraints and the key contribution is that the authors make these optimization layers efficiently differentiable. Networks with such layers can then in some sense “learn constraints” such as the constraints in a sudoku game to perform better. However, we entirely fail to see how differentiable optimization layers that can learn constraints would relate to our work as these are orthogonal to adding explicit manifold constraints to the trajectory obtained from integrating the learned gradient function. > I do not see comparisons to any of the above in the work you are doing, and it makes me hesitant to say that your algorithm works better than any existing ones. At the very least, a comparison needs to be made to these methods. I am willing to consider the point that your algorithm is more general but that does not exclude comparisons to these (and I'm still not convinced on the generality either). To sum up our rebuttal: We do not compare to these methods because, despite the apparent similarities, they do not apply to our setting. The few cases that do apply to some of our considered settings (primarily works on learning Lagrangian and Hamiltonian dynamics, that we have mentioned in the related work and also included comparisons where applicable) are limited to special cases and are therefore definitively less general than our method. If any aspect of our work or this response remains unclear, we would be very happy to discuss this further with the reviewer. **References** [1] Boumal, Nicolas, et al. "Manopt, a Matlab toolbox for optimization on manifolds." The Journal of Machine Learning Research 15.1 (2014): 1455-1459. [2] Boumal, Nicolas. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. [3] Xiao, Nachuan, et al. "CDOpt: A Python Package for a Class of Riemannian Optimization." arXiv preprint arXiv:2212.02698 (2022). [4] Smirnov, Oleg. "TensorFlow RiemOpt: a library for optimization on Riemannian manifolds." arXiv preprint arXiv:2105.13921 (2021). [5] Baddoo, Peter J., et al. "Physics-informed dynamic mode decomposition." Proceedings of the Royal Society A 479.2271 (2023): 20220576. [6] Richter-Powell, Jack, Yaron Lipman, and Ricky TQ Chen. "Neural conservation laws: A divergence-free perspective." Advances in Neural Information Processing Systems 35 (2022): 38075-38088. [7] Lezcano-Casado, Mario, and David Martınez-Rubio. "Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group." International Conference on Machine Learning. PMLR, 2019. [8] Kiani, Bobak, et al. "projUNN: Efficient method for training deep networks with unitary matrices." Advances in Neural Information Processing Systems 35 (2022): 14448-14463. [9] Donti, Priya L., David Rolnick, and J. Zico Kolter. "DC3: A learning method for optimization with hard constraints." arXiv preprint arXiv:2104.12225 (2021). [10] Amos, Brandon, and J. Zico Kolter. "Optnet: Differentiable optimization as a layer in neural networks." International Conference on Machine Learning. PMLR, 2017. [11] Gemici, Mevlana C., Danilo Rezende, and Shakir Mohamed. "Normalizing flows on riemannian manifolds." arXiv preprint arXiv:1611.02304 (2016). [12] Bose, Joey, et al. "Latent variable modelling with hyperbolic normalizing flows." International Conference on Machine Learning. PMLR, 2020. [13] Rezende, Danilo Jimenez, et al. "Normalizing flows on tori and spheres." International Conference on Machine Learning. PMLR, 2020. [14] Lou, Aaron, et al. "Neural manifold ordinary differential equations." Advances in Neural Information Processing Systems 33 (2020): 17548-17558. [15] Ben-Hamu, Heli, et al. "Matching normalizing flows and probability paths on manifolds." arXiv preprint arXiv:2207.04711 (2022).
Unfortunately, we believe there may have been a mistake when posting the previous responses and they may not have been visible to the reviewer. If this was indeed the case, we hope that they are visible now and that the content can be taken into account when making the final decision.
Summary
This paper proposes a general framework to stabilize NODE models. By introducing a new stabilization term, it can learn vector fields that satisfy various conditions. The authors evaluate the effectiveness of the proposed method on several physical systems represented by ODEs.
Strengths
- This paper proposes a new stabilization term that makes it possible to learn vector fields that satisfy various types of conditions. - The proposed method has been theoretically and experimentally evaluated, and the results are valid. - The paper is well written.
Weaknesses
- Lack of clarity on how the problem setup differs from HNN and LNN. - Insufficient discussion of applicability to PDEs.
Questions
- When considering conservation laws as constraints, does the proposed method require the energy function to be given explicitly, and does it also require the total energy to be known? HNN can learn a conserved system even when the energy function is unknown by incorporating a symplectic structure into the NN. Can you clarify the difference in problem setup (especially in input data) between HNN or LNN and the proposed method? - In section 5, application to PDEs is mentioned. Are there any difficulties other than the issue of computational complexity? Also, is it possible to consider boundary conditions in PDEs as stabilization terms?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
As written in Questions, it is better to be clear about the problem setting and add a discussion about the scope of application of the proposed method.
Summary
The authors propose a new method called stabilized neural differential equations (SNDEs) that enforce arbitrary manifold constraints for neural differential equations. The proposed method is applicable to any type of ODE, hybrid models (UDEs), and can incorporate any type of manifold constraints. The authors' showcase their work on first- and second-order systems including non-chaotic and chaotic, non-autonomous and autonomous examples.
Strengths
Originality: The related work is nicely covered with a specific focus on Hamiltonian/Lagrangian and other constrained neural differential equation methods. It has been made clear out of the related work, which previous approach is most similar to the work presented and how the current work differs/builds upon it. The proposed work is a novel perspective/approach on the existing work on constrained neural DE methods. It is largely based on earlier PhD thesis by Chin (1995) on stabilization methods for simulations of constrained multi-body dynamics. Quality: The submission is technically sound. I have not analyzed Proof 2 in depth, but based on the information that the authors' have provided, it seems supported. Clarity: The submission is very clearly written making it enjoyable to read as it is clear and understandable for the reader. The submission is well organized, the notation is clear, the function input/output dimensions are clarified, as well as matrices are defined.
Weaknesses
Quality: Despite providing a thorough overview of related work that also incorporate constraints, experimentally the authors only compare to NODE and SONODE which are known not to extrapolate well beyond the training regime due to the limited constrains/bias for the differential function/vector field. If the authors would have included comparisons to some of the other works mentioned in the related works section, such as Finzi et al., or other method that include a Hamiltonian constraint (SymODEN etc), it would put the present work in a better comparison to the related work in the field, as it provides an alternative to it. As such an explicit comparison has not been made, it is hard to evaluate the present method in context of other, constraint based Neural ODE type, methods. Clarity: Eq (3). I assume there is a missing $\in \mathbb{R}^n$ Significance: The proposed method is a relevant contribution to the field, however, I miss a better empirical comparison to other recent constrained neural ODE methods, this would make the submission a lot stronger and a better contribution to the existing work in the field.
Questions
The proof of Theorem 2, is this the author contribution or as the Theorem 2 comes from the work of Chin, 1995, was the proof already established then? For Eq (9), this states that the derivative is either 0 or negative, meaning that the vector field is decreasing or stable (0), how does this then relate to the example given in Fig. 1 where at around coordinates (0,0) the vector field is 'expanding'? 4.1., 4.2., 4.3., 4.4., 4.5 what are the test settings? what is the $\gamma$ value used in these experiments? In section 3 you mention that it might be beneficial to start training without stabilization, do you test this in practice? In general, you mention that 'are easy to use in practice across a wide variety of settings with minimal to no tuning', however based on results on double pendulum it could be suggested that as any hyperparameter also $\gamma$ requires tuning, meaning that the performance is sensitive to it. Have you done additional experiments to comment on whether there are certain experimental set-ups that require a certain $\gamma$ value, or a general advise what is a good starting value etc, which would be informative to the reader as a starting point? Are there additional experiments available that also compare the present method to existing constrained neural ODE models? At the moment, this is a major limitation of the presented work, which negatively affects the quality of the paper, even if the proposed method is sound.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have addressed limitations, in particular, as a key limitation it is stated that at the current implementation the method is limited to ODEs.
Summary
This paper aims to learn ODE dynamics where the dynamics has some extra equation constraint. For example, a double pendulum with energy conservation. In the problem setting, a neural network is used to learn the time derivative of the system, the dynamics is then rolled out with a classical solver. To ensure the rollout respects the conservation laws, the authors propose to add a stabilizing term to the time derivative, which can be proven to make the rollout asymptotically converge to the manifold defined by the conservation law. The stabilizing term is implemented with peudo-inverse of the Jacobin of the conservation function. Experiments show that the proposed method can stabilize the rollout w.r.t. the conservation law, compared to the method without stabilization.
Strengths
- The paper has good theoretical analysis and the practical implementation is based on the theories. - The experimental results are improved.
Weaknesses
- The baseline method involves no stabilization. Is there other simple method that can ensure the stabilization? For example, since the conservation equation is known explicitly, can one use one degree of freedom only to satisfy the constraints? I.e., if the output is $m$-dimensional, the network can only predict $m-1$ dimensions and use the conservation equation to compute the last dimension directly.
Questions
- The method involve computing the pseudo-inverse of the Jacobin matrix of $g$. If $g$ itself is complicated, e.g., contains higher order derivatives, would the computational cost become very significant?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
None
Summary
The authors of this paper proposed to augment neural ODEs with an additional stabilization term that is derived from the pseudo-inverse of a Jacobian matrix to learn dynamics of a system from data. The authors provided theoretical grantees to show that their augmented version is still capable of learning the original dynamics despite having an additional term and that the additional term actually stabilizes the learned dynamics. Experimental results were shown and compared with a vanilla NODE across a variety of dynamical systems with different characteristics.
Strengths
The overall paper, description and explanations of the work is very well written and relatively accessible even to a reader who is not very familiar to the topic. I also appreciate the simplicity of the idea, in that the proposed stabilization term could potentially be easily combined with other variations of neural ODE methods to achieve potentially more accurate predictions. The experimental results were presented in a wide variety of scenarios and the benefits of the proposed method does seem significant when compared to Vanilla ODEs.
Weaknesses
The following are the two improvements that I believe could improve the paper. 1. The idea behind this paper seems simple and the results seems promising, which given it's simplicity, could really be an impactful contribution. Hence, if the authors could also demonstrate the gains of this method on another variation of vanilla NODE and also perhaps an actual application, I believe this paper will be significantly stronger. 2. The overall paper is relatively accessible to readers not familiar with the field from a conceptual point of view, however, it'll be more informative if the authors could more candidly explain the connection between certain parts of the paper. For example, it is not very clear from an initial pass how does the manifold relates to the loss function and training of the NDE and the additional stabilization term.
Questions
1. Could the authors discuss further if there are potentially other candidates for the matrix F besides the ones that the author's used? 2. From reading the main paper alone, it wasn't very clear to me if the results shown were from the training trajectories or a set of testing trajectories with unseen initial conditions. I later realized that the information was provided in the supplementary material but I believe it at least be briefly stated in the main paper. 3. In Figure 2, what does (SO) NODE represent? 4. I noticed that the length of the trajectories and the resolution of each time step is sometimes different across the different experiments. Is there specific reason for this? 5. From what I understand, each trajectory is split into 3 time steps of non-overlapping chunks and then divided into training and validation. I am curious how does this method perform if the training and validation is not randomized, but is only trained on the initial segment of the trajectory and tested on the later segment of a trajectory?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Yes
Acknowledgment of rebuttal
I acknowledge that I've read the authors' responses to my comments and would like to follow up with the following points > We thank the referee for motivating us to compare our method more generally but are unsure what they mean by “another variation of vanilla NODE”. However, we have now added two additional baselines: 1. a HNN applied to the two-body problem experiment, and 2. a manifold ODE, based on a spherical tangent projection operator, applied to the rigid body rotation experiment. We hope these additional baselines make the paper stronger in the eyes of the reviewer and we emphasize again that we are not aware of any existing method for constraining neural ODEs that is applicable across all of our examples. I would like to clarify my comment on this point. What I meant by "another variation of vanilla NODE" was actually referring to augmented variants of the NODE model, such as those mentioned by the author in the first paragraph of the related works section. Since the authors claimed that all of these variants can, in principle, be stabilized by the proposed method, it would make the paper more convincing if the authors could also demonstrate that their proposed method is complementary to the augmented versions as well. > This is certainly an interesting question that we will investigate and report on the results. I look forward to these additional results and would especially be curious to see if the proposed stabilization term would degrade the extrapolation abilities when compared to vanilla NODEs
Further Experiments and Clarification
> I would like to clarify my comment on this point. What I meant by "another variation of vanilla NODE" was actually referring to augmented variants of the NODE model, such as those mentioned by the author in the first paragraph of the related works section. Since the authors claimed that all of these variants can, in principle, be stabilized by the proposed method, it would make the paper more convincing if the authors could also demonstrate that their proposed method is complementary to the augmented versions as well. We thank the reviewer for the clarification. We have now implemented augmented neural ODEs (ANODEs) both with and without stabilization and we’re happy to report the results here. We have also updated our Figures to now include (SO)ANODE with and without stabilization. While we can no longer upload figures, we will describe the results in detail. 1. **Vanilla ANODEs outperform vanilla NODEs but generally underperform SNDEs.** For the two-body problem experiment, vanilla ANODEs remain stable and even competitive with SNDEs in terms of relative error. For the rigid body and converter experiments, on the other hand, vanilla ANODEs remain stable and accurate for longer than vanilla NODEs but also diverge eventually. In this sense, vanilla ANODEs produce results that are intermediate between vanilla NODEs and SNDEs in terms of both relative error and stability. We thank the reviewer for this suggestion, which provides an additional interesting baseline against which to compare SNDEs. 2. **Stabilization interfaces seamlessly with ANODEs.** For the two-body problem experiment, stabilized ANODEs perform slightly better than SNDEs, while for the other experiments, stabilized ANODEs produce results essentially equivalent to SNDEs. This verifies our claim that other variants of neural ODE can be easily stabilized by the proposed method. If the reviewer agrees that the paper is improved by these additional baselines, and in particular by the verification of the flexibility of the proposed method, we would be happy if the reviewer would consider increasing their score. > I look forward to these additional results and would especially be curious to see if the proposed stabilization term would degrade the extrapolation abilities when compared to vanilla NODEs We apologise for the confusion here and we would like to further clarify the procedure for training and testing. In general, the trajectories used for training, validation, and testing are completely separate and differ in their initial conditions (and therefore also the value of the constrained quantity, e.g. energy). That is, *individual trajectories are not split* for training, validation and testing. Regarding the interesting question of extrapolation, we would like to draw the reviewer’s attention to Figure 5(c). This experiment differs from the others and is designed specifically to test extrapolation. In particular, each model is trained on a single short trajectory and then tested on a longer, unseen portion of the trajectory beginning at the last training observation. In essence, this experiment seeks to answer the question, “given a limited snapshot of a chaotic system, how well can we learn the system and predict its future dynamics?”. The results demonstrate that SNDEs vastly outperform vanilla NODEs on this task.
Acknowledgement of further experiments
I would like to thank the authors for the additional experiments and clarifications. Based on the additional results, I am happy to raise my score to a 7. Please include the additional results in the camera-ready version.
I would like to thank the authors for the additional experiments made and for answering all the questions voiced. As the authors have performed additional comparisons with respect to vanilla HNN and a manifold ODE with a spherical tangent projection operator, and the authors still obtain better or on par performance with their method I will increase my score to 6.
Thanks for the authors' rebuttal. I am not entirely familiar with the related literatures. From the paper's presentation I believe the proposed method is principled and works as desired. I am curious about the motivation of the proposed solution. In particular, why do we need to apply learning methods for such problems and are methods applicable in realistic scenarios? E.g., are the constraints always known?
We thank the reviewer for their interest. The Beucler et al. paper is well motivated in the context of applications such as numerical weather prediction and climate modeling, where it frequently occurs that fundamental conservation principles are known even while the exact dynamical laws are either not well understood (e.g. cloud microphysics) or not resolved numerically (i.e. important dynamical procceses occur below the grid scale) or both. As in the Beucler et al. paper, such conservation laws often turn up in the form of balance equations. For example, when modeling subgrid-scale convection, the rate of change of the total mass of water in a given atmospheric column must be balanced by the rates of evaporation and transpiration at the surface. We believe our method could be easily adapted to modeling such dynamics. More generally, while useful constraints might not always be known in advance for all learning tasks, we see great value in encoding known constraints or fundamental conservation laws as inductive biases wherever possible. Indeed, encoding physical laws as inductive biases continues to attract significant attention in the scientific machine learning community, see for example [1,2,3,4] in addition to the Beucler et al. paper. The main contribution of our work is to show how this can be done specifically in the context of constraints on neural ODEs. [1] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations." Journal of Computational physics 378 (2019): 686-707. [2] Rackauckas, Christopher, et al. "Universal differential equations for scientific machine learning." arXiv preprint arXiv:2001.04385 (2020). [3] Sturm, P. O. and Wexler, A. S.: Conservation laws in a neural network architecture: enforcing the atom balance of a Julia-based photochemical model (v0.2.0), Geosci. Model Dev., 15, 3417–3431, 2022. [4] Thomas Readshaw, W. P. Jones, Stelios Rigopoulos; On the incorporation of conservation laws in machine learning tabulation of kinetics for reacting flow simulation. Physics of Fluids 1 April 2023; 35 (4): 047103.
Reply
Thank you for your answer. I now have a better understanding of this paper. In the final version, please add careful explanations, especially about the differences in problem setup from HNN and LNN, and use cases of the proposed method. Also, unlike HNN and LNN, I think the fact that the proposed method can be used for arbitrary coordinate systems is a big advantage and should be emphasized.
Decision
Accept (poster)