Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximations

Answer part 1

Dear Reviewer, Thank you very much for the time you took and the comments you bring up that will help to improve our paper. ## Weaknesses ### Presentation (1) Concerning your first comment, could you name what you mean by “the layout of the paper is somewhat disorganized”? Do you mean the structure (i.e. Introduction; Background, Method, Applications, Discussion, Conclusion), or do you mean how/where we inserted the figure? What caused your comment “The presentation is unclear without a main thread”? Do you think that two sentences in the Background section would help to understand why we explain NeuralODE and VAE there, apart from the first two sentences of the abstract? Which complex concepts do you think are not explained adequately (dimensional reduction with VAE?), which concepts are unnecessary? (2) - - > answer in section questions (3) We used the ICLR template and did not adjust spacing between paragraphs and sections. Could you tell us at which position you see insufficient spacing? ### Experiments (1) First, I would like to reiterate the motivation for the ICLR conference “What is the significance of the work? Does it contribute new knowledge and sufficient value to the community? Note, this does not necessarily require state-of-the-art results. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc).” We believe that a big advantage of our work compared to other model-order methods (such as eDMDc) is the methodology itself; for learning (linear) surrogate models, we do not need to pre-determine bottleneck dimensions or choose appropriate basis functions. As it is a model-order reduction methods, its performance outcome is always a compromise between state reduction and prediction accuracy. I would like to explain why we cannot provide a comparison to the machine learning task of time series forecasting as in Koopman-based and VAE-based model you mentioned, e.g. in [1] and [2]. In our terminology, this would translate to predicting a time-series starting at an initial state $x^z(t_0)$, without control inputs $u(t), t>t_0$ and parameters $p$. In contrast to that, we aim for the generation of model-order reduced dynamical system models, that can predict the quantities of states $x(t)$ and output variables $y(t)$ of dynamical systems, given an initial state $x(t_0)$ and continuously arriving inputs signals $u(t), t>t_0$. This means, that forecasting of a time-series is not our target, our goal is building a dynamical system model. You might have noticed that we identify the initial latent state $x^z_0$ in equation 4ba just from the state vector $x(t_0)$. We obtain this state vector from the simulations of the physical model we aim to build a surrogate model for. We can do this for simulations; but not for real-world data. Control theory uses the term observability if one can determine the systems state. Observability does not mean that all states can be inferred from a snapshot of measurements at one time $t_0$. In general, important states might not be measured and can only be determined from a time-series $t_{-T_{obs.}:0}$ of system measurements $y(t)$. In control theory, state observers such as Luenberger observers and Kalman filters are therefore used. However, as we currently aim with B-NODEs for surrogate models that replace equation-based simulation models with neural-based models, we did not focus on developing a state observer. Additionally, we do also believe that the better scientific approach is to first develop a good method to learn dynamical models (potentially B-NODE). After that, we also figured that projecting the classical control theory concepts (Luenberger observer or Kalman filter) on the B-NODE context would be a certainly interesting area for future research. Then, a publication about that should for sure include a comparison to other baseline models for time-series forecasting e.g. as in [1,2] Unfortunately for now, no benchmarks are established for our targeted problem. Our method targets dynamical systems with inputs, whose models can be integrated by using an ODE solver and therefore can profit e.g. from adaptive-step size solvers to improve simulation speed and can be integrated industry relevant simulation framework such as Modelica, Dymola, Simulink etc.

Answer

Dear Reviewer, Thank you a lot for your time and the valuable feedback. Regarding the weakness you outlined, can you provide maybe one or two examples for what you mean? From my point of view, our method is applicable to every system that exhibits dynamics that are possible to be represented in a continuous form, i.e. $\dot{x}=f(x,u)$. ## Questions As space in the paper is limited, we tried to keep the $\beta$-VAE explanation short and referred in line 140 to [1], and mentioned the $\beta$ values implications again in line 314 and 346. You are right in your assumptions. A large $\beta$ will increase the importance of transmitting the information with the least amount of information possible (i.e. lowest number of states), while a small $\beta$ value makes the pressure on a reduced-order representation less strong. This can be seen in Figure 5, where small beta values correspond to more states, but also less prediction error. If we set $\beta=0$, the B-NODE degrades to a NODE on a latent space without any control of the latent space’s structure. By this we mean that the KL-Divergence in the loss forces the latent state for $\beta$-values $>0$ numerically to be centered around 0. Actually, when plotting a large number of states from simulations of a trained B-NODE, we can see a normal distribution of states around 0. For $\beta=0$, the model will bring the variance values $\sigma$ to 0, and the degraded B-NODE will use all states without limitations, maybe leading to instabilities as the same system state might have different latent representations. We will try to add a section with more clear discussion of the implications of different values for the $\beta$ values. We hope that this could clarify your concern. If any questions remain, we are happy to answer them. If not, we would be happy if you could increase your rating. Thank you! [1] Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in β-vae, 2018. URL https://arxiv. org/abs/1804.03599.

Updated paper version available

Dear Reviewer, We just uploaded a revised version of our paper and would like to point your attention towards changes we made to adress weaknesses you outlined: - To better evaluate B-NODE's performance on different systems, we added a benchmarks in Table 1 for the nonlinear case an in Table 2 for the linear case. In addition to the stratified heat flow model, we added a dataset from the MuJoCo (the MuJoCo Ant), a fluid system example (water hammer), and a CFD example (small grid). We also added comparision to other state of the art methods, apart from state space neural ODE and Latent ODE, for the linear case (eDMDc and DMDc). - We added more explicit explanation on the $\beta$-parameter, e.g. by adding the "direction" how $\beta$ scales the balance at multiple places in the text, e.g. line 147, 191. This influence can also be seen in Figure 4(a) and our added benchmarks in table 1 and 2. Motivated by your question what happens if we set $\beta=0$, we also added an experiment with $\beta=0$. We thank you again for your feedback and hope that we can further improve your impression of our work with the improvements made. We would be happy if the additional changes we made result in an improvement of your score. We remain open for discussion or questions if any remain. Thank you! Best, the authors.

Answer part 1

Dear reviewer, Thank you, a lot, for your detailed feedback and the time you put into that. I believe that the points you outlined will help to increase our paper’s quality. First, considering all of the reviewers’ answers, I want to address a point that seems important to stress more and that we will outline better in updated version of the paper as well. In our work, we’re learning a model that can react to continuous-time input signals $u(t)$, in which it differs considerably from in machine learning commonly known models for time-series prediction. These methods infer some kind of state $x(t_0)$ at a time $t_0$, and then predict output signals $y(t)$ for $t>t_0$. In contrast, we learn surrogate models for physical simulation models, of that we can access the initial state $x_0$ from simulation data, such that there is no need to infer an initial state from a sequence of variables $x_{-T:0}, y_{-T:0},u_{-T:0}$. Then, by continuously incorporating the input signals $u(t)$ at $t>t_0$, we predict the systems quantities $x(t), y(t)$ at $t>t_0$. Concerning your suggestion for presentation (1), we will therefore update the paper with more examples to make the targeted use-case more graspable. In the following, I would first like to react to the weaknesses you outlined, and then answer your questions. ## answer to weaknesses ### technical weaknesses Concerning your point evaluation, I would first like to explain why we can not provide an evaluation with method and datasets as in your cited papers [1] and [2]. While writing our paper, we were aware of their work (as we did also cite it), but their methods and our methods aim for different things: [1] and [2] concern time series forecasting and filling gaps of irregularly sampled time-series. In our terminology, this would translate to predicting a time-series starting at an initial state $x^z(t_0)$, without control inputs $u(t), t>t_0$ and parameters $p$. In contrast to that, we aim for the generation of model-order reduced dynamical system models, that can predict the quantities of states $x(t)$ and output variables $y(t)$ of dynamical systems, given an initial state $x(t_0)$ and continuously arriving inputs signals $u(t), t>t_0$. This means, that forecasting of a time-series is not our target, our goal is building a dynamical system model. You might have noticed that we identify the initial latent state $x^z_0$ in equation 4ba just from the state vector $x(t_0)$. We obtain this state vector from the simulations of the physical model we aim to build a surrogate model for. We can do this for simulations; but not for real-world data. Control theory uses the term observability if one can determine the systems state. Observability does not mean that all states can be inferred from a snapshot of measurements at one time $t_0$. In general, important states might not be measured and can only be determined from a time-series $t_{-T_{obs.}:0}$ of system measurements $y(t)$. In control theory, state observers such as Luenberger observers and Kalman filters are therefore used. However, as we currently aim with B-NODEs for surrogate models for replacing equation-based simulation models with neural-based models, we did not develop a state observer. Additionally, we do also believe that the better scientific approach is to first develop a good method to learn dynamical models (potentially B-NODE). After that, we also figured that projecting the classical control theory concepts (Luenberger observer or Kalman filter) on the B-NODE context would be a certainly interesting area for future research. Then, a publication for time series forecasting is possible and should for sure include a comparison to other baseline models as in your mentioned publications [1,2]. Unfortunately for now, no benchmarks are established for our targeted problem. [3] provides a review for neural-network based models for dynamical systems. Our method targets dynamical systems with inputs, whose models can be integrated by using an ODE solver and therefore can profit e.g. from adaptive-step size solvers to improve simulation speed and can be integrated industry relevant simulation framework such as Modelica, Dymola, Simulink etc. For this, only NeuralODEs and Neural Controlled Differential equations (NeuralCDE) are presented in the review [3]. However, both do not perform model-order reduction.

Updated paper version uploaded

Dear Reviewer, We uploaded today our updated version and would like to draw your attention to changes we made, in that we adress weaknesses you outlined. - Concerning limited evaluation, we added more experiments and comparision to other state of the art methods for the nonlinear case in table 1 (line 424) and linear case in table 2 (476). For the nonlinear case, we also provided an example with a dataset from MuJoCo. - Concerning clarity on main advantage, we added a discussion on that and comparison to other methods in Section 6 (starting on line 486). We also added more details on the considerable increase of computation speed (32 times-102 times higher) in line 395-400, and information about the number of required solver ODE calls in Figure 6(b) below. Concerning the points about vague exposition, we made the following changes: - Explicit description of tuning knob $\beta$: We added the direction how $\beta$ scales the balance add multiple places in the text, e.g. line 147, 191. This influence can also be seen in Figure 4(a) and the added benchmarks in table 1 and 2. - Why non-hierachical prior? Explanation added in line 159-160. - Defintion of state "active-ness" and definition of KL Divergence marginals per latent channels: We moved an equation ((4)) from the appendix that defines $D_{KL,z_i}$ per latent channel and used this definition in the following text. To additionally show that this definition of active-ness holds, we added an experiments in the Appendix (p. 30), cited in line 354, that shows that we can transform the weakly defined information bottleneck into a "hard" one by masking all latent channels with $D_{KL,z_i}<0.1$. Overall, we made multiple changes improving lines 143-184, where we describe dimensional reduction with $\beta$-VAE Concerning your question: - We made our design choice for independence of input encoding from parameters explicit in line 211. Concerning your suggestions for presentation: - We added more examples to conceptualize the approached problem better in the motivation (line 33-39). - We improved font size in figures. We thank you again for your feedback that helped to significantly improve our work. We tried to adresss all the points you outlined and would be happy if you would adapt your assesement of our work. We remain open to questions and discussion if there are any points still open. Best, the authors

Answer to review

Dear Reviewer, Thank you very much for your work and your valuable feedback. Regarding the weaknesses you outlined, I agree that the presentation might feel brisk at some times, regarding the page limit of 9 pages. Neverltheless, we tried to fit in as much explanations of foundations as possible, instead of for example referring to the review in [1]. Regarding ablation on the architectural choices, I would like to add something here: With the information bottlenecks we force the model to learn a “minimal” and orthogonal latent representation of the system’s dynamics. By orthogonal we mean that VAEs can learn disentangled informations, in that each latent factor describes a different aspect of information [2,3]. We hypothesize that this avoids overfitting, as only the necessary information are transmitted, and additional information channels that might cause overfitting, cannot be used. However, we observed during training no overfitting, i.e. train, test and validation losses had always (approximately) the same value. Additionally, we performed a hyperparameter sensitivity analysis that revealed that the method is robust against different architectural choices (number of layers, hidden dimension, activation function, prescribed latent dimensionality). Furthermore, we did choose the same hyperparameters for alle distinct networks, because hyperparameter tuning shouldn’t be a focus of this work. Would you think that including this kind of information might increase the paper’s quality (having in mind the page limit)? To your questions: After reading the reviewers answers, we feel like it’s important to stress again the fact that we’re learning a model that can react to continuous-time input signals $u(t)$, in which it differs considerably from in machine learning commonly known models for time-series prediction. These methods infer some kind of state $x(t_0)$ at a time $t_0$, and then predict output signals $y(t)$ for $t>t_0$. In contrast, we learn surrogate models for physical simulation models, of that we can access the initial state $x_0$ from simulation data, such that there is no need to infer an initial state from a sequence of variables $x_{-T:0}, y_{-T:0}, u_{-T:0}$. Then, by continuously incorporating the input signals $u(t)$ at $t>t_0$, we predict the systems quantities $x(t), y(t)$ at $t>t_0$. (1) Because of that, first, I want to clarify, the Latent ODE [4] was in the original publication not thought for incorporating continuous-time inputs. To my knowledge, there are no related works use latent variables (with an information bottleneck prescribed through KL-divergence) for model order reduction. There are works that use latent variable models (e.g. simple (not variational) Autoencoder based models), e.g. as described in the review of [5, p.62], but the latent space dimension is an important hyperparameter to be chosen. (2) I think this is a great question, as it stresses an advantage of our method. Classical DMD was initially developed for generated reduced order models of simulations with very fine spatial distribution (as it is the case in CFD). For generating surrogates of simulations of systems without fine discretization, one needs to extend the dimensionality of the observable space (i.e. the variables obtained by simulation) with additional observations, generated by basis functions, such that that fitting a linear operator can be performed upon these variables [5, p. 75]. Basis functions can be sines, cosines, linear combinations, polynomials, or radial basis functions. However, finding the right basis functions is an additional aspect of surrogate model generation, and for our approach, something like this is not needed. I hope we could answer your questions. If something should be hard to understand or new questions are raised, please don’t hesitate to write again. Thank you a lot!

Updated Paper Version available

Dear Reviewer, We uploaded today our updated version and would like to draw your attention to changes we made, that also adressed weaknesses you outlined. Concerning weakness (2), we tried to improve the presentation clarity by reformulating and adding several sentences. Most significantly, we added 143-184 more detailed explanation of dimensional reduction with $\beta$-VAE. We also added a more detailed motivation starting in line 33. Concerning weakness (3), we added on the last page of the paper a small hyperparameter sensitvity study. As pointed out in your questions, we also compared our method to DMD and and addded a comparision to other methods in the discussion section starting in line 486. We thank you again for your feedback and hope that we can further improve your impression of our work with the improvements made. Best, the authors.