From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems

We sincerely appreciate your insightful comments and valuable suggestions. We kindly recommend that the reviewer carefully reads the "**General response**". In response to the individual comments, we have thoroughly considered your suggestions and addressed all concerns in a point-by-point manner as outlined below. We hope that the reviewer recognizes the efforts and improvements we have made in both the methodology and experimental aspects. ``` W1: Novelty may be limited due to existing work... ``` **Response**: Thank you for your suggestion, and we supplement this work as a reference. While both our work and theirs utilize Fourier analysis, our approaches differ fundamentally. Firstly, our study focuses primarily on a family of parametric ODE/PDE systems, which exhibit unique properties distinct from N-body simulations. Secondly, while they add the power spectral error to their loss function, our approach involves matching the gradient flow. Lastly, building on the powerful NODE framework, we propose an efficient and robust training strategy that is widely applicable, even extending to PDE systems. ``` W2: Regarding architecture of the whole system, it’s not clear to me how the feedback loop works... ``` **Response**: Thank you for your careful reading and valuable comments. The feedback loop illustrated in Fig. 1 can be interpreted as a form of data augmentation, integrating the dynamics data predicted by the neural network with the original data. This strategy significantly expands the number of sample points, thereby enhancing the precision of Fourier analysis. For a more in-depth procedural understanding, please refer to Fig. 1 in the main text and Algorithm 1 in the appendix.   Moreover, the use of gradient flow error as the loss function constitutes the focal point of our research. This choice is advocated due to the fact that in the classical NODE framework, utilizing prediction error as the loss function frequently leads to some challenges involving high computational costs and the propensity for finding local optima, rather than the global optimum. ``` Q1: In Sec. 1, the authors claim that Fourier analysis provides theoretical guarantees... ``` **Response**: Many thanks for your valuable comments. The theoretical guarantees of our method stem from Theorem 1 of the main text (proof in Appendix A.1), which provides conditions under which the gradient estimated via Fourier analysis consistently converges to the actual system gradient.   It is noteworthy that our approach retains its efficacy even in scenarios where these conditions are not strictly met. For instance, many systems exhibit non-periodic behavior, and Fourier analysis, by default, assumes a periodic extension of the input discrete data. This may result in certain estimation errors at the boundaries. In practical implementation, we bypass this issue through disregarding the error at a small segment of the boundary and utilize the remaining error as the loss function (refer to code for details). Furthermore, as mentioned in the Point 1 of the "**General Response**", Fourier analysis and neural network complement each other by matching gradient flows from two distinct perspectives: spectral decomposition and dynamical modeling. In a statistical context, even with certain systematic errors in gradient estimation, this combination allows for robust dynamical modeling. Additionally, our experiments indicate that our method delivers satisfactory performance in modeling tasks for systems that do not strictly adhere to Theorem 1. ``` Q2: What’s the general guidance on selection of the cutoff in approximation of spatial gradients of PDEs... ``` **Response**: Thank you for your comment. In fact, our experimentation includes all common partial derivatives up to fifth order, without any prerequisite knowledge of dynamical equations. Through the neural network, our approach can automatically learn the complex interaction among the spatial derivatives, system state, and parameter values.   As for the hyperparameters in GRF, similar to the articles on FNO and DeepONet methods, these hyperparameters are pre-established. And the variations within a reasonable range are acceptable for these parameters. Detailed configurations can be found in Appendix B and the code for data generation. ``` Q3: In the evaluation section, baseline methods are limited... ``` **Response**: Thanks for your instructive comment. In fact, our goal is to introduce a novel training strategy for NODE via flow matching, not to focus on achieving SOTA performances on various tasks compared with numerous baselines. Therefore, we employ the common baselines to validate the effectiveness of our proposed approach. For further reasons, please refer to the Point 3 of the "**General Response**". In addition, the methods like transformer, n-beats and deepar indeed perform well in some tasks of time series prediction, but it is not equipped to handle the parametric systems.

Dear Reviewer, We hope this message finds you well! Thanks again for your valuable comments. We kindly remind you that the Author-Reviewer discussion is coming to an end and we would like to know if there are any remaining changes or questions you would like us to answer. We're more than happy to address these issues (before the rebuttal deadline). Otherwise, could you take this opportunity to re-evaluate our paper and update your score accordingly? Thanks! Best, Authors

Thank you for your valuable comments and helpful suggestions on the proposed framework. We kindly recommend that the reviewer reads the "**General response**" for a comprehensive overview of our main changes and clarifications. We address each minor and major points raised in the review point by point as follows. We sincerely hope that the reviewer recognizes the efforts and improvements we have made, and reassess the presentation and contribution of our work. ``` W1: Clarity is really lacking... ``` **Response**: Many thanks for your careful reading and helpful advice. We have conducted a comprehensive review and revision of the paper to enhance the readability and rigor. In what follows, we provide some explanations regarding the 8 detailed questions you care about. Q1: In fact, our experimentation includes spatial derivatives of up to fifth order, but without any prerequisite knowledge of dynamical equations. Leveraging the power of neural networks, our approach can robustly and automatically learn the complex interactions among the spatial derivatives, system state, and parameter values. Q2: Traditional NODE methods typically utilize the multi-step prediction error for training. However, our training strategy is different. As demonstrated in Eq. (5), we employ the Fourier analysis on a segment of training data to generate the temporal gradient estimate, denoted as $\bf{\hat{h}}’(t_n)$ at all sample points. The gradient errors at all points within this data segment then serve as the basis for our training loss. Q3: Prior to the implementation of the data augmentation scheme, we fully train the neural network using the existing data to acquire an optimal proxy model of the system dynamics. Subsequently, we further refine our model by training it with augmented data, thereby improving the accuracy of our modeling. The detailed implementation process can be found in the code file "multi_training.ipynb". Q4: In fact, many systems exhibit non-periodic behavior, and Fourier analysis, by default, assumes a periodic extension of the input discrete data. This assumption may introduce estimation errors, particularly at the boundaries. In practical implementation, we bypass this issue by disregarding the error at a small segment of the boundary and utilize the remaining error as the loss function to update neural network (refer to code for details). Furthermore, as mentioned in Point 1 of the "General Response," Fourier analysis and neural network complement each other by directly matching gradient flows from two distinct perspectives: spectral decomposition and dynamical modeling. In a statistical context, despite potential systematic errors in gradient estimation, this combination enables robust dynamical modeling. Q5: We apologize for any confusion in our earlier explanation. After computing the spatial derivatives in Fourier space, we proceed to transform them back to real space using the Inverse Discrete Fourier Transform. This conversion is essential before feeding the data into the neural network for the computation of temporal gradients. Q6: In the training phase, we consider the temporal gradient error in the response to Q2. When evaluating testing data, our metric involves calculating the mean prediction error across all sampling points. The accumulation of prediction errors over time is illustrated in Figs. 3-5 of the main text and Figs. S1-S3 in the appendix. Q7: Our experimental results, detailed in both the main text and the appendix, highlight the superior long-term stability of our method compared to the classical NODE. This enhanced stability is primarily attributed to our approach of directly incorporating the gradient error across the entire data segment into the loss function. This inclusion significantly improves the numerical integration stability of the neural network. Q8: We sincerely appreciate your comment and have addressed the identified error.  ``` W2: It is unclear how the model performs over longer periods... ``` **Response**: Many thanks for your valuable comments. In fact, for complex systems, especially chaotic ones, almost all methods struggle to achieve long-term predictions. This is due to error accumulation through iterative predictions. Therefore, to assess various methods more effectively, appropriate extrapolation prediction times are set up for different systems. ``` W3: The baselines do not seem very competitive... ``` **Response**: Thanks for your instructive comment. In fact, our goal is to introduce a novel training strategy for NODE via flow matching, not to focus on achieving SOTA performances on various tasks compared with numerous baselines. Therefore, we employ the common baselines to validate the effectiveness of our approach. For further reasons, please refer to the Point 3 of the "**General Response**".   In addition, the method from paper [1] does indeed perform well in the NS system, but it is not equipped to handle the parametric systems.

Dear Reviewer, We hope this message finds you well! Thanks again for your valuable comments. We kindly remind you that the Author-Reviewer discussion is coming to an end and we would like to know if there are any remaining changes or questions you would like us to answer. We're more than happy to address these issues (before the rebuttal deadline). Otherwise, could you take this opportunity to re-evaluate our paper and update your score accordingly? Thanks! Best, Authors

We thank you for your valuable comments and helpful suggestions on the proposed framework. We kindly recommend that the reviewer reads the "**General response**" for a comprehensive overview of our main changes and clarifications. Regarding the individual comments, we carefully consider your suggestions and address all the concerns point by point as follows. We sincerely hope that the reviewer recognizes the efforts and improvements we have made, and reassess the novelty and contribution of our work. ``` W1: I have concern about the novelty of this paper: it is rather a combination of the flow matching framework for functional/time series data, with the closed-form velocity approximated by discrete Fourier transform. ``` **Response**: Thank you for your comment. It is important to highlight that our approach should not be simplistically regarded as a fitting of the closed-form velocity. In fact, there is no definite closed-form Fourier function for the systems with variable parameters. Our approach, rooted in the NODE framework, introduces a novel strategy for effective and robust dynamical system modeling. In our framework, we leverage both Fourier analysis and neural networks synergistically, matching the gradient flows from two distinct perspectives: spectral decomposition and dynamical modeling. This integrated approach enhances the robustness of dynamic modeling a statistical sense. Additionally, we conducted comprehensive experiments across diverse systems, demonstrating consistently satisfactory performance in modeling tasks. For a thorough exploration and clarification of the novelty and contribution of our method, please refer to Point 1 in the "General Response".  ``` W2: Since this leans on more methodological/empirical paper, I will comment more on the evaluation part. I do not think the authors have done a thorough literature survey... ``` **Response**: Thanks for your instructive comment. Indeed, there are numerous methods for modeling dynamical systems and predicting time series, each possessing distinct strengths and weaknesses in different application scenarios. In our work, our goal is to introduce a novel training strategy for NODE via flow matching, not to focus on achieving SOTA performances on various tasks compared with numerous baselines. Therefore, we employ the common baselines to validate the effectiveness of our proposed approach. For further reasons, please refer to the Point 3 of the "**General Response**". In addition, we did not consider the methods requiring additional system information or deep learning methods that rely on extensive training data as baselines. For example, the PINN method [1] necessitates the incorporation of dynamical equations for the computing the loss function, rendering it incompatible with the data-driven modeling objectives in our study. Regarding the score-based diffusion models, we acknowledge that these models, as purely data-driven black-box models, can exhibit commendable performance in certain time-series prediction tasks. However, our hypothesis that the underlying systems can be modeled by differential equations with time-varying parameters, and the proposed method and baselines considered in this study are more suited to these characteristics. Therefore, our proposed framework can be viewed as grey-box models, demanding less training data than deep black-box models, while concurrently offering enhanced the interpretability of the underlying dynamics. ``` W3: To continue on the empirical evaluation, I do not understand why the authors did not include benchmarks on some of the realistic datasets... ``` **Response**: Thanks for your careful reading and instructive advice. In this work, we primarily focus on introducing a novel approach for efficient and robust dynamical system modeling. In line with many method-centric investigations, such as DeepONet, PDE-Net, and some extensions of NODE, we primarily conduct preliminary validation on synthetic dataset. To further substantiate the potential applicability of our method in real data, we conducted an additional experiment on a real-world system. Detailed information can be found in the Point 2 of the "**General Response**" and Appendix C.4.

Dear Reviewer, We hope this message finds you well! Thanks again for your valuable comments. We kindly remind you that the Author-Reviewer discussion is coming to an end and we would like to know if there are any remaining changes or questions you would like us to answer. We're more than happy to address these issues (before the rebuttal deadline). Otherwise, could you take this opportunity to re-evaluate our paper and update your score accordingly? Thanks! Best, Authors

We would like to thank the reviewer for the overall positive feedback and helpful suggestions. We revised the paper carefully according to the reviewer’s comments. Our major changes and clarifications are listed in “**General response**”. ``` Q1: Have you conducted an ablation study for the augmentation strategy? ``` **Response**: Thank you for your comment. We have undertaken the ablation experiments as detailed in Section 4.3, and the comprehensive results are presented in Figure 5 of the main text. Notably, Figs. 5(b)-(e) demonstrate that the data augmentation training strategy significantly improves the modeling performance without requiring additional data samples. Furthermore, an ablation study on the newly incorporated real-world experiment has also been carried out, and further details can be found in Appendix C.4.  ``` Q2: What was the reason behind introducing the control functions? Does it simply add to the complexity of potential tasks? ``` **Response**: Many thanks for your valuable comment. In practical applications, it is essential to acknowledge that the underlying systems commonly involve time-varying control (external) parameters. Investigating such systems not only enhances the complexity of modeling tasks but also brings the modeling process into closer alignment with real-world scenarios. Of course, our method can be naturally applied to systems without control parameters. ``` Q3: Is the method applicable if the requirements in Theorem 1 are not satisfied? ``` **Response**: Thank you for your insightful comments. Indeed, when Theorem 1 is satisfied, the gradient estimation from Fourier analysis can consistently converge to the system's actual gradient, thereby enhancing the learning of underlying system dynamics. It is noteworthy that our approach maintains effectiveness even in scenarios where Theorem 1 is not strictly met. For instance, many systems exhibit non-periodic behavior, and Fourier analysis, by default, assumes a periodic extension of the input discrete data. This can result in certain estimation errors at the boundaries. In practical implementation, we disregard the error at a small segment of the boundary. The remaining errors are then utilized as the loss function to update the parameters of the neural network, a process detailed in the accompanying code. Furthermore, as highlighted in the Point 1 of the "**General Response**", Fourier analysis and neural network complement each other by directly matching the gradient flows from two distinct perspectives: spectral decomposition and dynamical modeling. In a statistical context, despite potential systematic errors in gradient estimation, this combination allows for robust dynamical modeling. Additionally, our experiments indicate that our method delivers satisfactory performance in modeling tasks for systems that do not strictly adhere to Theorem 1. ``` Q4: Have you tried to apply this algorithm to real-life time series that are sampled from an unknown equation? ``` **Response**: Thanks for your careful reading and instructive advice. In this work, we primarily focus on introducing a novel approach for efficient and robust dynamical system modeling. Aligning with numerous method-centric investigations, such as DeepONet, PDE-Net, and various extensions of NODE, we primarily conduct preliminary validation on synthetic data. To further substantiate the potential applicability of our method, we conducted an empirical experiment on a real-world system. Detailed information can be found in the Point 2 of the "General Response" and Appendix C.4.