Chaotic systems make long-horizon forecasts difficult because small perturbations in initial conditions cause trajectories to diverge at an exponential rate. In this setting, neural operators trained to minimize squared error losses, while capable of accurate short-term forecasts, often fail to reproduce statistical or structural properties of the dynamics over longer time horizons and can yield degenerate results. In this paper, we propose an alternative framework designed to preserve invariant measures of chaotic attractors that characterize the time-invariant statistical properties of the dynamics. Specifically, in the multi-environment setting (where each sample trajectory is governed by slightly different dynamics), we consider two novel approaches to training with noisy data. First, we propose a loss based on the optimal transport distance between the observed dynamics and the neural operator outputs. This approach requires expert knowledge of the underlying physics to determine what statistical features should be included in the optimal transport loss. Second, we show that a contrastive learning framework, which does not require any specialized prior knowledge, can preserve statistical properties of the dynamics nearly as well as the optimal transport approach. On a variety of chaotic systems, our method is shown empirically to preserve invariant measures of chaotic attractors.
Paper
References (58)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This paper mentions that MSE loss is insufficient for learning the solution operator for chaotic dynamics. Two approaches are proposed to solve the problem. One is to use sinkhorn loss, and the other is to use contrastive learning. The authors evaluate the effectiveness of the proposed methods through experiments on several physical systems.
Strengths
- The problem of learning solution operators for chaotic dynamics is a challenging and interesting problem, and the authors have an interesting point of view (preserving invariant measures). - To my knowledge, the introduction of physics-informed optimal transport and contrastive learning is a new approach.
Weaknesses
- Although the problem setting is important and interesting, the proposed method seems to be a straightforward combination of existing technologies. - There is room for improvement in the writing of the paper. - There is no reference to previous studies on autoregressive approach.
Questions
- I understood that the prediction in equation (3) is the same as the autoregressive PDE solver proposed in [R1]. It would be good to add such previous studies. - Since the autoregressive approach performs forecasting iteratively, it seems to me that the numerical error increases in the long-term forecasting setting. Could you add a discussion on this point? Also, is it possible to forecast at arbitrary time intervals $\Delta t$ during training and testing? - I think it would be easier to read if the description of the previous studies (optimal transport and contrastive learning) and the proposed method are separated. In section 3, I feel that the novelty of this paper is not described clearly. - There are other ways (e.g., Maximum Mean Discrepancy) to measure the distance of a distribution besides optimal transport. It would be good to clarify the motivation for using optimal transport, and the same goes for contrastive learning. [R1] Johannes Brandstetter, Daniel E. Worrall, Max Welling, Message Passing Neural PDE Solvers, ICLR, 2022.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
It would be good to discuss in detail the advantages and disadvantages of using Optimal transport and contrastive learning.
Summary
This paper proposes two methods to preserve invariant measures of chaotic systems in the multi-environment setting when training neural operators. Given some expert knowledge of the underlying dynamical systems, they propose a new optimal transport loss, which uses this knowledge to match the statistics. Without expert knowledge, they use a contrastive learning approach to learn invariant statistics, which are used to construct the feature loss to preserve these invariant statistics. These two methods show better performance to reproduce the invariant statistics of noisy chaotic systems.
Strengths
1. Long-time prediction of chaotic systems is different, this paper proposed two methods to better represent chaotic systems using neural operators by preserving the invariant statistics of chaotic attractors. 2. The methods are robust to noise and chaos.
Weaknesses
1. These two methods are aimed at different problem settings: the optimal transport-based approach needs prior knowledge, while the contrastive learning approach does not; the optimal transport-based approach seems not to depend on the multi-environment setting, whereas the contrastive learning approach does depend on this setting. These two methods are relatively separate, and the relationship between them is not close. 2. The two methods are mixed together, so the structure of the paper does not look clear. 3. It seems that the optimal transport-based approach requires too many equation constraints. For example, in the two examples given in the paper, almost all of the terms in the underlying dynamical equations are used. 4. As for the contrastive learning approach, higher requirements are put forward for the diversity of data. 5. The introduction of the Noise Contrastive Estimation (InfoNCE) loss is not described clearly in the main text.
Questions
1. Is it possible to provide the problem setting, limitations, and advantages of the two methods more clearly, and clearly explain the difference between the two methods, instead of introducing them together? 2. Whether the performance can be maintained if more limited prior knowledge is utilized for the optimal transport-based approach? 3. If all those equation constraints have to be used to ensure performance, should it be compared with methods such as physical informed neural operator that utilize similar information instead of the conventional neural operator? 4. As for the contrastive learning approach, what kind of diversity does the data need to meet to ensure better performance? Could you provide an intuitive explanation?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The authors have adequately addressed the limitations.
Summary
The authors proposed a training framework to preserve preserve invariant measures of chaotic attractors. First, they identify training standard neural operators using MSE on chaotic dynamics does not work. Then they suggested to train neural operators to preserve the invariant measures of chaotic dynamics and the time-invariant statistics. Importantly, they proposed two approaches to train neural operators to preserve those statistics. One is the optimal transport-based approach, and the other is the contrastive learning approach. They empirically showed that both approaches capture the true invariant statistics.
Strengths
The paper is well written and organized. The paper follows a well-defined structure, with a logical flow of ideas throughout.
Weaknesses
The experiment part could benefit from more substantial content. Comparison with previous methods should be addressed in the experiment part. See below for the two questions as well.
Questions
What is the advantage of preserving the invariant features in chaotic dynamics? Does it really contribute to long-term forcasting, which suppose to be the objective? If it does, I expect the author to show an improvement in long-term forcasting quatitatively by adopting one of the two proposed approches. The proposed method is evaluated on the synthetic data, the Lorenz-96 and the Kuramoto–Sivashinsky chaotic dynamics. I am wondering how the model performs on some real world data, e..g, the climate system.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors have addressed the limitations
Summary
The paper uses neural operators to track the invariant statistics of chaotic systems. It novelly proposes to use the optimal transport loss and contractive learning to match the distribution, so that the learned model correctly track the attractor among the chaotic behavior. The paper uses FNO as a backbone to study the Lorenz and KS equation. Experiments show the proposed methods better track the behaviors such as histogram and spectrum.
Strengths
The paper proposes to learn the dynamics of chaotic system with the optimal transport loss and contractive learning. The proposed methods are more stable compared to the standard MSE/L2 loss, and they can capture the attractor of the system. Experiments show these methods better track the statistical behaviors such as histogram and spectrum.
Weaknesses
The paper only studied Lorenz and 1d Kuramoto–Sivashinsky equation. It will be very interesting to study how these methods scale to 2d Navier-Stokes (Kolmogorov Flow) problem. It will be interesting to discuss time complexity.
Questions
I have a few questions: - Which method (Sinkhore algorithm and contrastive learning) does better? What are their comparative advantages and disadvantages? - How does the training time of the Sinkhore algorithm and contrastive learning? I assume they will be slower than the standard supervised learning. It will be great to report the runtime and training time. Would the wasserstein loss be easier to optimize compared to L2 loss? It will be interesting to show a training curve. - How does the proposed methods compared to generative model [1, 2]. For example, in Generative adversarial neural operators they also matches the wasserstein distance. [1] Rahman, Md Ashiqur, et al. "Generative adversarial neural operators." arXiv preprint arXiv:2205.03017 (2022). [2] Lim, Jae Hyun, et al. "Score-based diffusion models in function space." arXiv preprint arXiv:2302.07400 (2023).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The paper addressed the limitation.
Summary
This work considers the problem of learning invariant statistics of chaotic dynamical systems. It is motivated by the observation that training neural operators by standard MSE loss focuses on short-term predictability and thus may miss an attractor’s properties. Two ways of augmenting the MSE loss for capturing invariant stats are discussed: One based on the Wasserstein distance between prob distributions across explicitly provided stats (physics-informed), and one based on contrastive feature learning. Experiments on chaotic benchmarks, the Lorenz-96 and the Kuramoto-Sivashinsky equations, are shown.
Strengths
The paper is timely and addresses an important issue, namely how we can capture crucial properties of an observed dynamical system beyond just short-term predictions. In general I like the approach, and think this could be a very fruitful addition to the current state. In general, the paper is also well written, and the ideas well motivated. Although incorporating invariant stats into the loss has recently been discussed, this paper offers a different perspective on this topic.
Weaknesses
On the other hand, the experimental evaluations are, in my view, quite weak, and also a lot of related and relevant literature is not discussed. The experiments primarily show that on any of the evaluation measures that particular method wins, for which exactly that measure has been included or accentuated in the loss function: If the loss is only MSE, the only-MSE trained method wins on short-term predictions; if the loss includes invariant statistics, the method wins on exactly those invariant stats. This is not surprising, but would be expected for almost anything explicitly included in the loss term (i.e., the method for which a specific property was included in the loss should have an edge when evaluated on that particular property). It would have been much more convincing if methods which include invariant statistics could also reproduce many *other* properties of the attractor that were *not* explicitly included in the loss, like its Lyapunov spectrum or fractal dimension. Besides, two of the columns in Table 1 \& 2 don’t contain any indication of statistical error and are therefore somewhat meaningless in my mind (are the differences significant or negligible?). To see how the methods perform on at least one empirical chaotic problem would have also been nice. Literature-wise, the idea of including invariant statistics in the loss has been considered by a number of recent publications, for instance https://arxiv.org/pdf/2304.12865.pdf or https://pubs.aip.org/aip/cha/article-abstract/33/6/063152/2900453/Learning-dynamics-on-invariant-measures-using-PDE?redirectedFrom=fulltext. The fact that purely MSE-based methods have difficulty capturing invariant stats for chaotic systems has recently also been extensively discussed in https://proceedings.neurips.cc/paper_files/paper/2022/hash/495e55f361708bedbab5d81f92048dcd-Abstract-Conference.html and, relatedly, https://arxiv.org/abs/2306.04406, although these authors arrive at different conclusions of how to tackle the problem. My perception is that this problem has received a lot more attention recently than the authors’ related works section makes one believe.
Questions
My primary concern is really the weaknesses in the experimental evaluations as listed above. This sect. was a bit disappointing to read after the exciting start of the paper and presentation of methods.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
4 excellent
Limitations
Yes.
Thank you for your detailed replies, which cleared up most of my confusion. The added experiments explain the performance of the model in the statistically limited case, making the discussion more systematic. Based on this, I increased the score accordingly.
Thank you for your reply!
Thank you for your constructive feedback of our paper and appreciation of our work! We will make sure to include the performance of the OT using limited statistics and other discussed clarifications, in the revision of our paper.
Thank you for answering my questions
Re Q1.1: I think there is still a large gap between 'the average number of hurricanes per year' and predicting statistical features of the chaotic dynamics...To me, this model still cannot tell me the exact long-term prediction of chaotic dynamics. Some of the statistical features can be visualized by for example, PCA, t-SNE based on history data. Why do we need to use the model to tell us these low-dimensional invariant featuers? Re Q1.2: Can the authors give an example of the invariant feature of a REAL chaotic system? The Lorenz-96 is a synthetic system with a known low-dimensional feature. If the invariant feature of a real chaotic system is too complex, or too hard to define, how to verify the model can be applied to a real system? Generally I kind of agree with Reviewer GF1X that '...to see how the methods perform on at least one empirical chaotic problem would have also been nice...'
Response to your questions (1/2)
Dear reviewer, we greatly appreciate your prompt feedback. Please allow us to address the concerns you've highlighted. > Re Q1.1: I think there is still a large gap between 'the average number of hurricanes per year' and predicting statistical features of the chaotic dynamics… The “average number of hurricanes” is an example of a statistical feature of climate models. Of course, there are many other important statistical features that will be relevant for building a high-quality climate emulator. Our goal is to use these statistical features to help train better deterministic emulators that will reproduce the true underlying chaotic attractor. Moreover, our CL approach is precisely focused on automatically identifying a rich collection of informative statistical features without requiring prior knowledge. Our results show that training an emulator using the invariant statistics learned from CL significantly improves the ability of the emulator to preserve many physics-informed statistical metrics we use for evaluation. > To me, this model still cannot tell me the exact long-term prediction of chaotic dynamics. No model can even *in theory* tell us the exact long-term prediction of chaotic dynamics. This is because chaotic systems exhibit extreme sensitivity to initial conditions, i.e., any small change or noise in the initial condition results in an exponentially divergent solution, even if we use a mathematically exact model. That is why we focus on statistical features that characterize chaotic dynamics. During the evaluation, we use statistics-based metrics that characterize the chaotic attractor of our simulated systems and show that these indeed match the data well when performing long-term predictions: 1,000 time steps in the Lorenz 96 system and 500 time steps in the Kuramoto–Sivashinsky system. Moreover, lots of studies related to climate science have shown that preserving the statistical features of chaotic systems is very important in predicting and understanding the climate. Fundamental studies ([1, 2]) show that the dimension of attractors characterizes the minimum number of variables to describe the system. [3, 4] also demonstrate that preserving the invariant attractors is helpful for determining extreme events. A recent study ([4]) also identified rMSE as a problematic evaluation metric for climate prediction as it implicitly assumes two time series should be aligned for every individual day, which is nearly impossible for real data. Instead, they use an invariant summary statistic for evaluating climate models. > Some of the statistical features can be visualized by for example, PCA, t-SNE based on history data. Why do we need to use the model to tell us these low-dimensional invariant featuers? Yes, PCA, or t-SNE can be used to visualize the features. However, these features do not come for free, and our goal is not visualization. While PCA or t-SNE can visualize known features, these methods do not allow us to discover new invariant statistical features. We also do not specifically target low-dimensional or visualizable features. In fact, our CL approach allows us to learn general high-dimensional invariant statistics without prior knowledge, which can then be used for training emulators.
Response to your questions (2/2)
> Re Q1.2: Can the authors give an example of the invariant feature of a REAL chaotic system? The four features we adopted in the evaluation of our experiments have been widely used to characterize the chaotic systems in real applications, e.g. climate: 1. The joint distribution of physics-informed summary statistics is a common choice for evaluating the quality of emulators for complex chaotic systems ([5]). 2. The Fourier energy spectrum, which described the distribution of kinetic energy as a function of frequency, is a common high-dimensional invariant statistic used in the analysis of fluid systems ([6, 7]). 3. The leading Lyapunov exponent (LLE) is a widely adopted measurement that characterizes how fast the dynamical states diverge with respect to time ([2, 8]). 4. The fractal dimension ([1, 9]), as a measure of complexity, has been used to describe the spatiotemporal climate data. > The Lorenz-96 is a synthetic system with a known low-dimensional feature. If the invariant feature of a real chaotic system is too complex, or too hard to define, how to verify the model can be applied to a real system? Our proposed CL approach is exactly designed to deal with complex systems where there is no significant prior knowledge! While simulated, Lorenz-96 and Kuramoto–Sivashinsky represent *high-dimensional* chaotic attractors with origins as models for fluid turbulence, and our experiments show that our proposed approaches work well for training better emulators for such systems. We are not aware of a low-dimensional characterization of the Lorenz-96 or Kuramoto–Sivashinsky attractor. > Generally I kind of agree with Reviewer GF1X that '...to see how the methods perform on at least one empirical chaotic problem would have also been nice...' We would also love to evaluate on empirical data! However, as we discussed in our response to reviewer GF1X, there is very little publicly available empirical data for systems as complex as the spatiotemporal chaos shown in our experiments. We would love to see this change in the future. However, current spatiotemporal modeling papers are almost all evaluated on high-quality simulations. The one exception is weather data used for training emulators such as FourCastNet [10] and ClimaX [11], which are very large models that cost millions to train. We would love to see how well these approaches scale to such models but that is outside the scope of this work. Thank you again for your time providing the feedback! If there are any further questions or details you’d like to discuss, we are here to assist. We look forward to hearing back from you. [1] Estimating the Dimensions of Weather and Climate Attractors. Fraedrich, Klaus. (1986) [2] Predicting uncertainty in forecasts of weather and climate, Palmer, T N. (2000) [3] Universal behavior of extreme value statistics for selected observables of dynamical systems. Lucarini, Valerio, et al. (2011) [4] A locally time-invariant metric for climate model ensemble predictions of extreme risk. Virdee, Mala, et al. (2023) [5] Multiscale Simulations of Complex Systems by Learning their Effective Dynamics. Vlachas, Pantelis R., et al. (2021) [6] Characterization and prediction of runoff dynamics: a nonlinear dynamical view. Islam, M.N, Sivakumar, B. (2002) [7] Global energy spectrum of the general oceanic circulation. Storer, Benjamin A., et al. (2022) [8] Predictability of Weather and Climate. Krishnamurthy, V. (2019) [9] Estimating the Fractal Dimension and the Predictability of the Atmosphere. Zeng, X., et al. (1922) [10] FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators. Pathak, Jaideep et al. (2022) [11] ClimaX: A foundation model for weather and climate. Nguyen, Tung et al. (2023)
remaining points
I thank the authors for their response which clarified some of my points. A few remaining remarks: - W1: This doesn't really address my point, I think. My argument was that it's quite expected that if you optimize method 1 for property A, then method 1 will outperform others not optimized for A. The additional evaluations on LE and FD mostly address this point, however. - W2: Since this is all model-based, you have the Jacobians, so why can't you assess the whole LE spectrum? - W4 \& W5: The authors state as a key difference to [4,5] that these do not involve settings with noise. Didn't check this, but what I do recall is that [4,5] used several *empirical* datasets which I would assume contain a lot of noise! So this falsifies the authors' statements about both the noise and the unavailability of empirical datasets with characteristics of chaos.
Response to your questions (1/2)
Dear reviewer, we greatly appreciate your valuable feedback! Please allow us to address your comments as follows. > W1: This doesn't really address my point, I think. My argument was that it's quite expected that if you optimize method 1 for property A, then method 1 will outperform others not optimized for A. The additional evaluations on LE and FD mostly address this point, however. We agree with this point. To some extent, the OT method can be thought of as an “oracle” approach that lets us measure a best-case level of performance that we would aim for with a method that cannot be optimized for property A. This is why we chose to include performance on property A among our results. But the results are certainly strengthened with your suggestions of LE and FD. It is also important to note that our OT method also performs significantly better than the baseline when we evaluated the performance using Fourier energy spectrum, and the leading Lyapunov exponent in the high noise setting. In addition, new experiments with our OT method (also mentioned in our discussion with Reviewer DhyB) demonstrate that OT could also improve performance when using limited knowledge of the invariant statistics, which implies the better generalization ability of the OT approach. Last, we want to emphasize our CL approach, not designed explicitly on any evaluation metrics (with the absence of physics-informed prior knowledge), delivers superior performance than the rMSE baseline on all four metrics we evaluated on. > W2: Since this is all model-based, you have the Jacobians, so why can't you assess the whole LE spectrum? Thank you for your suggestion! We initially focused on obtaining LLEs since it is a much more straightforward calculation, but we have now obtained estimates for the full Lyapunov spectrum, as requested. We present the results regarding the Lyapunov spectrum in the table below. For the Lyapunov Spectrum Error, we report the sum of relative absolute errors across the full spectrum: $\sum_i^d |\hat{\lambda}_i - \lambda_i| / |\lambda_i|$, where $d$ is the dimension of the dynamical state (i.e., 60 for Lorenz 96). As suggested by [5], we also compare the number of positive Lyapunov exponents (LEs) as an additional statistic to measure the complexity of the chaotic dynamics. We compute the absolute error in the number of positive LEs $|\sum_i^d \mathbf{1}(\hat{\lambda}_i > 0) - \sum_i^d \mathbf{1}(\lambda_i > 0)|$ averaged over the test instances. |r| Training | Leading LE Error $\downarrow$ | Lyapunov Spectrum Error $\downarrow$| Total number of positive LEs Error $\downarrow$| |-------------|-------------|-------------|-------------|-------------| |0.1 |$\ell_{\rm rMSE}$ | **0.014** (0.006, 0.021) | 0.388 (0.110, 0.309) | 0.526 (0.000, 1.000) | |0.1 | $\ell_{\rm sinkhorn} + \ell_{\rm rMSE} $| 0.049 (0.040, 0.059) | **0.256** (0.168, 0.285) | 0.375 (0.000, 1.000) | |0.1 | $\ell_{\rm feature} + \ell_{\rm rMSE} $| 0.065 (0.058, 0.073) |0.285 (0.164, 0.289) | **0.365** (0.000, 1.000) | |||||| |0.2 | $\ell_{\rm rMSE}$ | 0.175 (0.156, 0.191) | 1.940 (0.522, 0.726) | 4.248 (4.000, 5.000) | |0.2 |$\ell_{\rm sinkhorn} + \ell_{\rm rMSE} $| 0.019 (0.006, 0.030) | 0.837 (0.122, 0.590) | **2.540** (2.000, 3.000) | |0.2 |$\ell_{\rm feature} + \ell_{\rm rMSE} $| **0.012** (0.006, 0.018) | **0.769** (0.138, 0.568) | 2.819 (2.000, 3.000) | |||||| |0.3 |$\ell_{\rm rMSE}$ | 0.446 (0.425, 0.463) | 1.979 (0.702, 0.939) | 7.230 (7.000, 8.000) | |0.3 |$\ell_{\rm sinkhorn} + \ell_{\rm rMSE} $| 0.101 (0.062, 0.134) | **1.186** (0.571, 0.745) | **4.824** (4.000, 6.000) | |0.3 |$\ell_{\rm feature} + \ell_{\rm rMSE} $| **0.067** (0.045, 0.091) | 1.290 (0.558, 0.780) | 5.603 (5.000, 6.000) | Table 1. **Performance with varying noise level on Lorenz-96 evaluated on Lyapunov spectrum.** The average (25th, 75th percentile) error rates over 200 testing instances of training the neural operator with (1) only rMSE; (2) Sinkhorn (OT) and rMSE (using prior knowledge to choose summary statistics); and (3) contrastive feature loss (CL) and rMSE (no prior knowledge used). In the presence of high noise, OT and CL give lower relative errors on the Leading Lyapunov exponent (LLE). When evaluating the full Lyapunov spectrum, OT and CL show significant advantages than the baseline. In addition, the lower absolute errors of the total number of the positive Lyapunov exponents (LEs) suggest that OT and CL are able to match the complexity of the true chaotic dynamics.
Response to your questions (2/2)
> W4 & W5: The authors state as a key difference to [4,5] that these do not involve settings with noise. Didn't check this, but what I do recall is that [4,5] used several empirical datasets which I would assume contain a lot of noise! So this falsifies the authors' statements about both the noise and the unavailability of empirical datasets with characteristics of chaos. We thank you for your further questions! As there is no [5] in the original reference list, we assume you are pointing to [3] and [4]. Our work focuses on training neural operators, like the Fourier neural operator (FNO), on fully observed high-dimensional deterministic chaos, usually in the form of spatiotemporal systems governed by PDEs, and not on general time-series data. The empirical datasets, such as the ECG and EEG datasets ([4]), are very much partially observed and potentially stochastic and so do not fall under the problem statement we describe in Section 2. Additionally, we should note that the empirical datasets in [3] and [4] are single-environment problems and cannot be used to test our multi-environment approaches. We would ultimately like to apply this approach to train emulators using empirical fluid dynamics data or global weather data, like the data used to train FourCastNet ([6]) and ClimaX ([7]), where the goal is to predict spatiotemporal climate dynamics for multiple subregions and time periods. We will make sure to clarify this in the final paper. Regarding the noise setup, you are right! [3] and [4] do test on empirical time-series data which is expected to contain noise. [3] states in their empirical study that they operate on de-noised and smoothed data after preprocessing the raw data ([8]), while [4] does not apply preprocessing. We will amend our discussion to reflect this in our final paper. However, on their simulated datasets, which are closer to our problem setup, [3] does not add any noise and [4] only adds up to 5% ($r=0.05$) noise, which is much lower than the noise in our experiments. Last, we want to emphasize that both [3] and [4] are only using the short-term rMSE loss, which is our baseline in the experiments, and perform poorly in chaotic setups. We thank you again for your time providing us with the feedback! We are happy to assist if there are any questions or details you would like to discuss. We are looking forward to hearing back from you. [1] Constraining Chaos. Platt et al, April 2023. [2] Learning dynamics on invariant measures using PDE-constrained optimization. Botvinick-Greenhouse et al, Chaos, June 2023. [3] On the difficulty of learning chaotic dynamics with RNNs. Mikhaeil et al, NeurIPS 2022. [4] Generalized Teacher Forcing for Learning Chaotic Dynamics. Hess et al, June 2023. [5] Constructing Discrete Chaotic Systems with Positive Lyapunov Exponents. Wang, Chuanfu, et al. (2017) [6] ClimaX: A foundation model for weather and climate. Nguyen, Tung et al. (2023) [7] FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators. Pathak, Jaideep et al. (2022) [8] Nonlinear noise reduction: A case study on experimental data. Kantz, H., et al. (1993)
one final point
I thank the authors very much for their thorough response. There is one last point that I’d like to see clarified though: The authors state that training using just rMSE will not do a good job on chaotic attractors. However, I brought up [3,4] precisely because in those papers only rMSE is used (plus some amendments to the training procedure), yet apparently superb recovering of chaotic long-term behavior is achieved. Thus, that these approaches perform poorly in chaotic setups is not true, it's in my understanding actually what they were designed for! I do think that the authors’ idea of loss criteria including long-term statistics is an important conceptual contribution, but apparently this is not the only route for recovering long-term statistics, even for highly chaotic systems; similar for [1,2] who augment the loss through LE/FD. A fair and balanced discussion should probably make that clear.
Response to your questions.
Dear reviewer, we thank you for your prompt feedback! Once again, we are sincerely thankful for your suggestions on evaluating the Lyapunov spectrum and fractal dimension. Please allow us to address your concerns for references as follows. > I thank the authors very much for their thorough response. There is one last point that I’d like to see clarified though: The authors state that training using just rMSE will not do a good job on chaotic attractors. However, I brought up [3,4] precisely because in those papers only rMSE is used (plus some amendments to the training procedure), yet apparently superb recovering of chaotic long-term behavior is achieved. Thus, that these approaches perform poorly in chaotic setups is not true, it's in my understanding actually what they were designed for! I do think that the authors’ idea of loss criteria including long-term statistics is an important conceptual contribution, but apparently this is not the only route for recovering long-term statistics, even for highly chaotic systems; similar for [1,2] who augment the loss through LE/FD. A fair and balanced discussion should probably make that clear. Thank you very much for bringing up these great points! We will make sure to add the discussion to the revision of our final paper. While there is not a precise apples-to-apples comparison due to differences in the problem setup, in the following, we summarize our current understanding of how these approaches relate to and differ from ours and speculate about future work. [1] provides an alternative choice of regularization using LE and FD, it would be interesting to test in combination with our work in the future and see how the method works on a high-dimensional and noisy chaotic system when LE and FD are hard to compute reliably ([5]). [2] brings a unique insight into using a Fokker-Planck PDE for modeling the probability state for selected dynamical variables. However, since it requires estimating a high-dimensional probability distribution using finite mesh grids, it is challenging to scale the method to our high-dimensional problem. The architecture (RNNs) and teacher forcing (TF) [3,4] focused on could not be applied to our multi-environment setup without *major* modifications. Specifically, they focus on single environment dynamics and prove the optimal time interval of TF should depend on the LLE of the dynamics (eq. 17 in [3]). However, each trajectory in our training and testing dataset has its own LLE since we are working on multi-environment dynamics, which implies the direct application of [3,4] would result in a suboptimal choice of the time interval of TF and might impact the performance. In addition, we speculate that the high-noise setup we focus on might lead to further differences in performance. We thank you again for bringing up these valuable references, and we are especially delighted to study these contemporary works and see the increasing attention in the field of learning the chaotic attractors! All these approaches are interesting approaches that lend insight into learning chaotic attractors using different techniques, all with relative advantages and disadvantages. Exploring the relative performance of these methods across settings would be a major experimental undertaking and beyond the scope of this submission, but it would be a fascinating study for future work and help us develop a more comprehensive understanding of different approaches. **References** [1] Constraining Chaos. Platt et al, April 2023. [2] Learning dynamics on invariant measures using PDE-constrained optimization. Botvinick-Greenhouse et al, Chaos, June 2023. [3] On the difficulty of learning chaotic dynamics with RNNs. Mikhaeil et al, NeurIPS 2022. [4] Generalized Teacher Forcing for Learning Chaotic Dynamics. Hess et al, June 2023. [5] Impracticality of a box-counting algorithm for calculating the dimensionality of strange attractors. Greenside, H. S. et al, Phys. Rev. A, 1982.
Thanks
Thank you. I understood the authors are considering a multi-environment setting, I’m not so sure though it makes such a big difference for the argument I was trying to make. Either way, I didn’t mean to imply by my remarks the authors necessarily benchmark against these other methods (although the high-noise setting would have been interesting given that some of the other methods excel even on noisy empirical data). I just wanted to point out that there are apparently different ways to achieve excellent long-term behavior on chaotic systems, even if one sticks to rMSE, and a balanced discussion and literature context should probably reflect that. This doesn’t take away from the authors’ own approach. I raised my score by 2.
Thank you for your reply and the fruitful discussion!
Dear reviewer, we are sincerely thankful for your constructive feedback, thoughtful comments, and appreciation of our work! Our fruitful discussion sheds light on the future work for learning the invariant attractors for chaotic dynamics. We truly value your feedback and will make sure to include the results of Lyapunov Spectrum and FD, as well as our valuable discussion of the related contemporary works, in our final paper. Thank you!
Reply
Thank you for your response. I now have a better understanding of this paper, and the experimental results using MMD are also useful. I agree that the idea of using OT and CL to predict chaos dynamics is new in itself. I will raise my score by 1.
Thank you for your reply!
Dear reviewer, we are sincerely thankful for your constructive feedback and appreciation of our work! We will ensure to include the experimental results using MMD, and discussions of our OT and CL approaches in the final paper.
Thank you for the discussion!
Dear reviewers and ACs, we are very grateful for your time and efforts in reviewing our paper and providing us with a fruitful discussion! We are excited to see that our reviewers appreciated our novel contributions to training better emulators for chaotic dynamics. Our results have been strengthened by the additional evaluation metrics and experiments suggested by several reviewers. We also had a very fruitful discussion that helped us clarify the motivations and the advantages of our approaches. We are also excited to see the growing interest in chaotic dynamics from contemporary works that will form the basis for future work in this area. We will make sure to include these additional results and discussions in our final paper.
Decision
Accept (poster)