Neural Ideal Large Eddy Simulation: Modeling Turbulence with Neural Stochastic Differential Equations

Summary

The authors introduce a neural SDE model for LES flow fields. The model structure is motivated by the ideal LES approach and the model learns a data-driven closure term. The learned latent representation captures the variability and fine-scale structure of the fully-resolved DNS solution that is lost by the LES filter.

Strengths

The paper is excellently motivated, presented and embedded into the theory. Stochastic modeling for turbulence as presented in this paper is an important direction to explore and therefore the paper would be valuable contribution to the field.

Weaknesses

1. The authors seem to use the term "closure model" more loosely than usual in the introduction. Out of the citations for data-driven closure models in line 39, only [45] defines a closure model as the term is used in the context of RANS and LES. 1. Lines 177-182 are slightly misleading as they can be understood to suggest that the model would learn possible DNS realizations as $Z_t$ instead of an opaque latent representation. (Actually, lines 211-213 clarify this but could be moved forward/merged into the first paragraph) 1. A figure showing the flow (and predictions) after, for example, 200, 400, etc. steps would help the reader appreciate how chaotic the flow is and how much the velocity field evolves over 800 steps. 1. A figure comparing the DNS and filtered data would also be helpful for the reader to understand the effect of LES filtering better.

Questions

1. Eq (2): Why can the pressure be eliminated? 1. Line 87: Which boundary conditions do you impose and how do you do so implicitly? 1. Is there a mix-up of $h_\theta$, $h_\psi$, $h_phi$, $g_\theta$ and $g_\phi$ happening in Eq (8), (9), (10)? 1. Eq (14): You could define $f$ more explicitly here. 1. Eq (16): What do you mean with spatio-temporal lifting? Does w correspond to z? What is $\mathcal{D}$? 1. What exactly is the relationship between the SDE solver timestep and the simulation timestep? Are the temporal intervals $\mathcal{T}_i$ rescaled to $[0, 1]$?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors discuss limitations adequately.

Summary

This paper introduces a data-driven method for approximating the closure term in Large Eddy Simulation (LES). The closure term represents the unresolved scaling effect caused by reducing the computational grid through downsampling. In comparison to the traditional physically-informed approach, the learned closure term has the ability to automatically capture relevant features without the need for specific domain expertise or manual design. The model improves temporal resolution by employing latent space evolution with stochastic propagation. The proposed model demonstrates competitive performance compared to the Filtered DNS reference across different wave numbers, and significantly outperforms the baseline model.

Strengths

- The proposed method outperforms several baselines in terms of the squared error and the accuracy of turbulent kinetic energy spectrum. - The latent space evolution implements the procedure to solve the SDE equation, which introduces the domain specific inductive bias into the model.

Weaknesses

- The model's training solely relies on a single reconstruction loss, which is atypical for stochastic latent variables. The absence of prior regularization with respect to latent space implies that the stochastic component added at each temporal step acts more like a sparsity regularization, as the model is trained to minimize the stochastic effect rather than explicitly capturing the physical dynamics. Apart from prediction accuracy, there is a lack of additional benchmarks to demonstrate the advantages of the stochasticity, such as prediction diversity. - When comparing the proposed niLES with deterministic Neural Network-based models, it should be noted that niLES introduces additional parameters $h_{\phi}$ and $g_{\theta}$ as well as more computational routes, potentially leading to an unfair comparison. The authors should showcase the model's performance while varying the magnitude of the Wiener process in Equation 15 and compare it to the scenario where the noise magnitude is zero. - The current evaluation protocol does not adequately assess the scalability of the model, particularly with respect to wave numbers. To address this limitation, the authors should exclude a certain range of wave numbers as test cases, enabling a more comprehensive evaluation of the model's scalability.

Questions

Q: It would be better if the authors could show the stochastic property in latent space brings diverse and valid results in the LES space.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This submission proposed a data-driven method to learn a.closure model to simulate the results from DNS. The key part is a latent stochastic process by Neural SDE. And finally compute the Monte-Carlo approximation.

Strengths

1. The model treats the DNS as a stochastic process, instead of a deterministic process as in many previous works. 2. Empirical results indicate it performs well on Kolmogorov flow.

Weaknesses

1. The number of datasets is small. 2. It lacks recent deep learning-based methods as baselines.

Questions

1. Since the work treats DNS field as stochastic, why can you compute RMSE in the figure 3? 2. Why do you design a stochastic process on latent space, instead of directly on original field?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See questions and weakness.

Summary

This paper targets learning turbulence closure models for RANS simulations via neural networks. The paper proposes to use a neural SDE on the latent space of a transformer to predict different samples from the distribution of the next state, and then compute an average over these. This process is unrolled and trained for a sequence of multiple steps.

Strengths

Overall, this is a good idea, and I'm not aware of a an NSDE being previously used in this form. Thus, I see the general direction of the paper and the promising approach as strong points.

Weaknesses

On the other hand, the paper targets a single, two dimensional Kolmogorov flow secnario as the only test case. In addition, only a single deterministic NN is compared to (plus an implicit LES solver). For this single data set, the paper is lacking a stable evaluation: multiple, differentily initialized models evaluated across multiple tests to obtain a stable result are not evaluated. In addition, NeurIPS is a very broad ML venue. Turbulence is definitively an exciting topic here, but even more important would be a broader evaluation, ideally with substantially different secnarios to show that the method has merit beyond turbulence. In its current form, I don't think that the results are sufficient for a NeurIPS paper. I see two ways to improve this aspect of the submission: either the authors focus their writing on the turbulence secnario (cf. below), and present multiple scenarios in this context, or non-turbulence cases are included to broaden the scope. A more stable evaluation with multiple models and tests should be included either way. This potentially also could help to show the benefits of the method more clearly. Right now the gains in terms of accuracy and the differences in the TKE spectrum seem to be mild. Additional cases could show areas where the approach gives larger improvements. In addition, I would also recommend that the authors include additional learned baselines. This is less crucial, but would nonetheless help to put the work into the context of previous methods at NeurIPS, ICLR & co. I also do want to mention two weak points in the writing. One is that I found the motivation (esp. L44) quite unintuitive: the "inductive bias of the LES field" is not very clear, and the summary implies this is "simply" a matter of choosing the right architecture. Things get clearer afterwards, but reading the paper front to back, I think this summarizing question is not helping a reader. The conclusions are also not a good fit with the rest of the paper: suddently, a transition is is made to generic chaotic systems. The whole previous paper targets a single specific scenario in the form of Kolmogorov turbulence, and presents a single set of results. Hence, this outlook is not supported by the content of the paper. I can understand that the authors have hopes that their method will at some point in the future generalize to other cases, but it should be made clear that this is an outlook. Rather, references to specific works where the authors see potential would be interesting to give here. Overall, I want to encourage the authors to continue their direction of work. Nonetheless, I find it difficult to directly argue for accepting this paper in its current form.

Questions

Which other, specific applications and scenarios do the authors see for their method?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are discussed briefly.

Rebuttal

I’d like to thank the authors for the comments and the updated results. The cylinder case is very good to see! In terms of evaluation, I would encourage the authors to provide an averaged evaluation across multiple trained models (with different random seeds) for a final version. Nonetheless, I’d be happy to support an accept for this paper, and I’ve raised my score.

Further questions

I appreciate the authors’ efforts to address my inquiries and share results related to neural ODE and the description of sample diversity. Having gone through the feedback provided by the authors, I have further questions regarding the results, particularly on the comparison to deterministic methods. (1) Concerning the training objectives, the introduced approach learns features in the latent space through VAE-like objectives. Yet, it outperforms deterministic methods when considering reconstruction error. This seems somewhat unexpected to me, given that a standard VAE often yields blurred results and struggles to retain high-quality details. (2) Referring to Figure 4, the deterministic NN (d) seems to generate a sharp output, but it concurrently leads to high-frequency artifacts. My impression is that these artifacts might be attributed more to specific model configurations and training issues rather than an inherent property of deterministic NNs. (3) When it comes to assessing the diversity of samples, how can one accurately evaluate their correctness beyond mere qualitative visualization?