Newton-Cotes Graph Neural Networks: On the Time Evolution of Dynamic Systems

Summary

This paper proposes Newton-Cotes Graph Neural Networks (NC), which are a time-continuous extension of Equivariant Graph Neural Networks (EGNNs). The authors also conducted a mathematical analysis on the predictive variance of NC under the Gaussian assumption. Through numerical experiments, the authors have demonstrated that the NC-formulated EGNNs outperform the baseline EGNNs.

Strengths

- The method is clear and can be generalized to other GNN classes beyond Equivariant GNNs.

Weaknesses

**The proposed method is a special case of a larger GNN class, but there is no comparison provided with this GNN class in the submitted paper.** Although the authors have demonstrated the effectiveness of the NC integrator in conjunction with GNN through numerical experiments, the paper fails to compare the proposed Newton-Cotes Graph Neural Networks (NC) with the most competitive baseline class, Graph Neural Ordinary Differential Equations (GDEs) [1]. It would be highly beneficial to compare the proposed method with the more well-established GDE models. In my opinion, the proposed NC can be considered a special case of GDE where the integrator is the Newton-Cotes method and the GNN corresponds to the proposed EGNN. Furthermore, based on the advancements in Neural ODE research, the limitation mentioned in section 3.5 could already be addressed using adjoint sensitivity methods. [1] Graph Neural Ordinary Differential Equations

Questions

- Regarding the selection of the numerical integrator, it would be beneficial to clarify the advantages of using NC over more well-known and commonly used integrators such as Runge-Kutta or Dormand–Prince. - When it comes to training NC+, is it necessary to collect the K-1 intermediate points?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

- The current manuscript lacks a comparison with the most competitive baseline methods, as well as a comprehensive review of relevant literature.

Summary

This paper builds on recent advances in deep learning for physical simulation to produce reliable results of long-term consecutive predictions. The authors provide a proof that equivariant graph neural networks learn a linear map from initial velocity to a target velocity, and propose to inject inductive bias based on Newton-Cotes (NC) formulas as alternative of standard equivariant graph convolution. The paper introduces techniques for training, such as a new objective and a new regularization loss for higher order features, and also gives theoretical guarantee that variance can be reduced by increasing the degree of NC formula. Experiments in various multiple body interaction systems demonstrate that the proposed model outperforms state-of-the-art baselines .

Strengths

The paper is appropriately placed in the current growing literature on scientific machine learning to create a novel fast model for tackling complex problems in the realm of particle systems. Although the proposed architecture is simple in appearance, the motivation of introducing NC formulas is convincing giving both concise visualization and theoretical claims on the limitation of the existing models. Results of the experiments show that introducing NC formulas is effective to perform long-term accurate prediction. In my opinion this is a good paper and above the acceptance threshold.

Weaknesses

Although the proposed methods NC(k) and NC(k)+ outperforms state-of-the-art baselines, it is still somewhat unclear which components of NC(k)+ help to improve the performance of NC(k)+. What if setting all the coefficients of Newton-Cotes formula to be 1 (or some constant) while remaining the other parameters same? Without some additional ablation study, it is a bit difficult to get a clear picture of how new components of the proposed method would help improving its prediction performance. In the forward process, how is the coefficients of Newton-Cotes formula obtained? Is the coefficients trainable or derived from the equation (15)? It is also unclear how to derive coordinate at intermediate time step $t^{k}$ once velocity is updated by the formula (13), while the formula (12) is used only when computing the final state at T. For training time shown in Figure 2, why does the training cost of NC with GMN grow much faster than NC with the other backbones? It will be helpful to get more insight of the trends. Also, the order of NC(k) seems to be set K<=8, due to the catastrophic Runge’s phenomenon. Are there any limitation on the length of time interval [0, T] to which new NC points are spaced? How does the length of the interval affects the performance?

Questions

Please have a look at the weakness mentioned in the strength and weakness section and address these. Overall the idea is interesting, the authors need to substantiate their claims in light of possibly a few more experiments. **Minor question** What does “steps” of Table 4 in Appendix C mean?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Yes, the limitations are discussed in the appendix.

Summary

This paper integrates Newton-Cotes formulas from numerical analysis in (equivariant) graph networks. The authors introduce an approach that makes several velocity estimations to predict the integration towards the final state of a system. Further, they show that standard equivariant graph networks can be interpreted as a trivial case of their formalism. The proposed method (NC) has provably lower errors/variance, and becomes better with more velocity estimations. NC can be used with supervision from intermediate velocities, but it can also work well without supervision. The authors demonstrate the effectiveness of their method on several benchmarks, and using multiple graph network architectures as a backbone.

Strengths

The paper is very well-written and easy to follow. The proposed method is well-motivated from numerical analysis, and it is simple and clear. The proposed method shows improvements on 3 different benchmarks, and using 3 established equivariant graph networks as a backbone. The non-supervised version of the method (NC) can be directly integrated in many existing graph network architectures.

Weaknesses

The analysis on the prediction errors of intermediate velocities (figures 3, 6, 7) could be more in-depth, especially for the unsupervised version of the method (NC). It is unclear why the errors often increase as training progresses, or why they tend to fluctuate erratically, and whether this has an impact on performance, or if the model can make up for it. The qualitative visualizations, especially for MD17 and n-body (figure 8) are very hard to interpret.

Questions

1. Following the weaknesses listed above, in figures 3 and 6, why do the errors fluctuate or keep increasing with epochs? What is its impact on performance? 2. The Newton-Cotes formulas exhibit a reflection symmetry. Is it possible that the model is learning to predict the reflected intermediate velocities? An ablation study on that would be useful. 3. In subsection 4.6, the authors perform an ablation study on the method's ability to make consecutive predictions. There are related works that propose equivariant graph networks for multiple step prediction, including LoCS [2] and EqMotion [1]. It would be interesting to examine the applicability and added value of NC to such methods. As an example, it would be interesting to see the integration of NC in LoCS, since this work treats velocities in a principled manner and makes predictions in local frames. #### Minor 4. In lines 161-162, the authors mention that "EGNN always uses the same input velocity and coordinate as constant features in different layers". However, EGNN does use the updated coordinates in different layers, at least in the n-body setting. #### References [1] Xu, Chenxin et al. EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning. CVPR 2023. [2] Kofinas, Miltiadis et al. Roto-translated Local Coordinate Frames for Interacting Dynamical Systems. NeurIPS 2021.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations of their work.

Summary

The paper introduces a new method for predicting system dynamics. Unlike past works that predict the average velocity between start and end time, the proposed approach makes several velocity estimation at different time points and then uses Newton-Cotes formula for computing the integration. This method reduces the prediction error and improve accuracy in predicting future state of the system. The paper improves this method further by incorporating loss for each predicted velocity estimate, and shows better performance than current SOTA systems.

Strengths

1. The paper is clearly written with good figures. Supplementary material provides additional GIFs and proofs to help with understanding. 2. The paper proposes an interesting method to improve the current dynamics prediction systems by predicting velocity estimates at different time points and then using Newton-Cotes method to integrate them. Paper proofs that this method leads to reduction in variance, and therefore the error. NC+ approach further reduces the error by incorporating additional loss. 3. The paper shows that even their NC method outperforms the previous SOTA approaches significantly. 4. Despite predicting velocities at different time points, this method doesn't degrade the training time a lotl.

Weaknesses

I feel the novelty of the method is limited. The method replaces the average velocity estimate with multiple velocity estimates that are then integrated over, something that seems pretty obvious. Also, something that is not clear is why the performance improvement saturates at only K=2, as shown in Fig 2? Another weakness is that the inference time scales linearly as the value of K is increased.

Questions

I would like authors to address the points raised in weakness section. What is the intuition behind accuracy not increasing past K=2. It seems that higher K should result in even better performance, but that's not the case.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have not addressed limitations in their work.

Additional questions

Thank the authors for the response. Most of my concerns were addressed. I have a few follow-up questions on the following response: >Regarding the order of NC (k), the neural model learns to compensate for the prediction error (as illustrated in Fig. 1 and Lines 36-44). Setting a larger k may reduce the variance of the prediction error and make the optimization easier, but this strategy does not always work especially when the model is robust. Furthermore, the accuracy gain exponentially decreases as we estimate more points in numerical integration. Take Fig. 1 as an example, a shift from 1e-1 to 1e-2 yields a more significant gain compared to a shift from 1e-3 to 1e-4. For >Setting a larger k may reduce the variance of the prediction error and make the optimization easier... my question is the impact on prediction error when making the interval length longer, say [0, T] to [0, N*T]. Since the order of NC(k) (and NC(k)+) is bounded from above due to the catastrophic Runge’s phenomenon, the prediction error would also be bounded from the bottom. It would be helpful if the authors provide insights or theoretical/experimental results that clarify the robustness of the proposed method against the interval length. > Take Fig. 1 as an example, a shift from 1e-1 to 1e-2 yields a more significant gain compared to a shift from 1e-3 to 1e-4. What do you mean by shift? In appearance, I don’t see any terms relating to shift in figure 1. Can you expand the explanation?

Official Comment by Reviewer MNf4

I would like to thank the authors for their rebuttal; they have addressed my questions. I am very positive towards this paper and I think its contribution is significant. I am keeping my score to an 8. I hope the authors include the related works on multi-step prediction in the camera-ready version, alongside any experiments performed with the proposed method on top of them.