Modulated Neural ODEs

Summary

This paper proposes a modulated neural ODE (MoNODE) that can separately model the dynamics and time-invariant modulating variables (e.g., parameters of dynamics or contents of agents). Such modulating variables are modeled by averaging the embedded trajectories of given observations (i.e., the outputs of modulator prediction networks for a observed trajectories) with respect to the time, similar to the tranformation-invariant (TI) pooling approach. The proposed approach can be integrated into various NODE models simply. The authors validate the proposed MoNODE for various dynamical systems, including some synthetic examples (sinusoidal, Lotka–Volterra Hamiltonian, bouncing ball), images dynamics (rotated MNIST), and CMU motion capture dataset. The experimental results demonstrate the proposed MoNODE is better than NODE in terms of the interpretablility well as long-term predictability of dynamics. *** This paper proposes a concept of separating and learning dynamics and time-invariant modulator variables, and I believe it could be beneficial for various time-series prediction problems in scientific and engineering fields. However, I have doubts about whether the proposed model maintains the time-invariance property explicitly. Additionally, there are some concerns regarding comparisons with similar papers and validation in more complicated chaotic systems. Therefore, I currently consider this paper to be at a borderline reject.

Strengths

- The proposed approach is simple and principled, making it easy to integrate into various NODE-type models. - The paper is well-written and easy to follow. - The experimental results demonstrate that the learned latent dynamics of MoNODEs are more convincing and interpretable compared to those of NODEs. This may be attributed to the explicit decomposition of the time-invariant contents of agents from the dynamical behavior itself.

Weaknesses

- The time-invariance of the moduator prediction network is only gauranteed for the finite and fixed time horizon of $T_e$. As a result, it is not entirely clear whether the proposed model can effectively capture the time-invariant aspects of observations over an arbitrary time span. - Similarly, the proposed model requires the observation of length $T_e$ to predict the time-invariant modulating variables. Therefore, it is not greatly applicable for the case when $T_z < T_e$. - The paper lacks a comparison with a relevant previous work [1], which factorizes the latent space into content and motion variables and models the dynamics using the (Hamiltonian) latent NODE. *** [1] Khan, A., & Storkey, A. J. Hamiltonian latent operators for content and motion disentanglement in image sequences. NeurIPS 2022.

Questions

- I think it would be great to experimentally demonstrate that a trained modulating prediction network indeed guarantees time-invariance in the long-term time horizon. For example, it would be helpful to plot a graph to visualize the change in $d$ or $s$ over time (ideally to be a constant line). - Is the performance of the trained MoNODE still good when $T_z = T_e \ll T_{train}$? It seems that an ablation study is needed to check this. - I am curious whether MoNODE would be applicable to chaotic dynamics. For instance, one could consider the parameters ($\sigma$, $\rho$, $\beta$) of the Lorenz system as dynamic modulating variables. Can MoNODE effectively learn and infer them? - Max pooling [2] is commonly used as the pooling function for invariant mappings (e.g., $s^n = \max s_{i}^{n}$ for the static modulator case). Is there a specific reason for using averaging? It would be nice if the authors could conduct a comparison between max and average pooling invariant maps. *** [2] Laptev, D., Savinov, N., Buhmann, J. M., & Pollefeys, M. Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. CVPR 2016.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors mention some limitations (limited to deterministic systems, generalization to out-of-distribution modulators, ...) of the proposed method in the Discussion section. Another limitation worth mentioning is the weakness section of my review: not entirely guaranteed time-invariance, limited application when $T_z < T_e$, and the missing relevant reference.

Summary

This paper proposes a framework for Neural ODEs called Modulated Neural ODEs, which combines the conventional latent Neural ODEs with two Modulator Prediction Networks that are used to extract the time-invariant static and dynamic modulator variables, respectively. Their empirical results demonstrate that this framework can improve generalizations to new dynamic parameterization and long-time forecasting, and can better capture the true unknown factors of variation as well.

Strengths

1. The proposed MoNODE framework has better capability of generalizing to new dynamic parameterizations and long-term forecasting. 2. They experimentally verify that the MoNODE framework is able to better capture the true unknown factors of variation with modulator variables.

Weaknesses

1. Experimental examples are not convincing enough. The dynamical systems chosen in this paper are relatively simple and naive. For more complex systems such as trajectories generated by a family of chaotic systems controlled by different parameters, it is doubtful whether the MoNODE framework can still extract effective modulator variables. 2. Though NODE as the baseline group maintains the same dimension as MoNODE, additional parameters are still introduced in MoNODE, such as parameters in the Modulator Prediction Networks. It is not clear whether the slight improvement in performance is due to the additional number of parameters. 3. This paper does not directly provide a visual presentation or intuitive explanation of the static and dynamics modulator variables extracted by the modulator prediction networks.

Questions

1. When comparing with NODE, could you provide a comparison of the number of parameters and set up a more reasonable experiment to prove that the performance improvement is not caused by the increase in the number of parameters? 2. Whether the MoNODE framework can outperform the conventional NODE on the task of learning more complex dynamical systems, such as trajectories generated by a family of chaotic systems controlled by different parameters? 3. For the CMU Mocap datasets, due to the addition of the modulator variables, the MoNODE is supposed to significantly improve the performance of NODE in the Mocap-Shift task, but in fact, the improvement effect is not obvious, which is far less than that of Mocap. Can you further explain this phenomenon? 4. Could you directly provide some visual description and intuitive interpretation of the modulator variables extracted by the modulator prediction networks?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

The paper proposes an approach for generalizing NeuralODE to different system parameters, in the setting of a single dynamical system with different parameters. A dynamic and static modulators are applied to specific parts of the encode-process-decode architecture: the dynamic modulator acts on the NeuralODE part, while the static modulator only acts on the decoder. They are learned in an end-to-end manner. Evaluation is performed on synthetic datasets, as well as a real dataset (walking humans), by using different NODE variants. It showcases the generalization capabilities of the 2 modulators. The paper’s idea is interesting, and I found the experiments are convincing.

Strengths

- The proposed framework is easy-to-integrate to existing NODE methods to increase their performance. - The latent space is more interpretable than with vanilla NODE, and provides high correlation wrt the dynamics parameters - The proposed method drastically improves the used NODE methods for long-range prediction

Weaknesses

-

Questions

- Is there a way the modulations spaces could be used for interpolation between different systems? - Could this work be extended to process PDE equations?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The work only studies dynamical systems up to 3 dimensions, would it be able to scale to higher dimensions?

Summary

The authors introduce Modulated Neural ODEs by incorporating a static modulator that averages observation embeddings across the whole time and a dynamics modulator that averages observation embeddings by time subsequences. The proposed model consistently improves the existing model's ability to generalize to new dynamic parameterizations.

Strengths

The presentation is pretty straightforward and the idea is simple yet effective, as shown by experiments.

Weaknesses

- As the output of the static modulator and the dynamics modulator is used by concatenation, I would like to see the experimental results without the static modulator or the dynamics modulator to determine if they are both essential. - I would like the authors to add the inference time cost comparison with and without those modulators.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have already discussed the limitations of this work, e.g., the current exploration is only limited to deterministic systems and the performance of the work is largely affected by the base model’s performance.

Summary

Existing neural ordinary differential equations (NODEs) present the problem of variation across trajectories only through initial state values. As a solution to the problem, Modulated Neural ODE (MoNODE), which improves the existing NODE method by distinguishing dynamic and static states, is presented. In MoNODE, 1) static modulators and 2) dynamics modulators are used to divide dynamic variables from time-invariant variables. By attaching these two modulators to existing NODE studies, they show good performance in various datasets.

Strengths

1. They presented the problem of handling static variables well in NODE, and suggested dynamics modulator and static modulator as a solution. This is very intuitive. 2. It is very attractive that it can be applied to any NODE model and does not have to be divided into static and dynamic variables in advance. 3. They showed the performance using various datasets to evaluate dynamic and static modulators.

Weaknesses

1. They experimented by dividing 5.1 dynamics modulator variable and 5.2 static modulator variable data to evaluate their modulators, but they could not show whether each modulator actually played its role.

Questions

In experiments 5.1 and 5.2, static variable and dynamics variable data were separately tested to evaluate each static modulator and dynamics modulator. Was the dynamics modulator also applied to static variables in these experiments? If the dynamics modulator was used even in the static variable experiment, you should remove the dynamics modulator. If you did not use dynamics modulator in static variable experiment, you should experiment using dynamics modulator. In other words, through these experiments, it is necessary to check whether each modulator is playing a role.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is no limitations

Summary

The paper propose to improve long term forecasting abilities of NODEs by incorporating the dynamics with learnt static parameters. The system is composed of a static modulator and a dynamic modulator, the final solution is sampled from a distribution conditioned on the static modulator and the dynamic generated using the dynamic modulator. Empirical results suggests improvements over NODE on time simple series datasets.

Strengths

Presentation is clear, and the paper is well written. The empirical improvements on time series data is significant on the long run.

Weaknesses

Similar work has been proposed, see https://arxiv.org/abs/2302.13262 for an example. Although the methods are slightly different, the fundamental idea of incorporating with static parameters are the same. There are no clear theoretical guarantees that the method will work better. The network structure is very complicated, the reviewer is worried about the training difficulties. It would be helpful to see some ablation and show it is easy to train.

Questions

How are the networks trained? There are so many different networks. Also, how is this method significantly different from existing literature?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

1 poor

Limitations

N/A