## 3. Here we add the discussion of the references
> References: Discussing other developments such as
> - Brandstetter et al., CLIFFORD NEURAL LAYERS FOR PDE MODELING -> usage of multivector representations together with Clifford convolutions. The authors show the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surrogates by their Clifford counterparts on 2D Navier-Stokes tasks
The Clifford neural layers [1] use the Clifford algebra to compute multivectors, which improves the effiency for simulations involving multiple fields, and they provide Clifford-FNO implementations as an extension of FNO. While in this work, we only considered the pressure field so Clifford may not be helpful, it will be interesting to explore using Clifford-FNO within our GINO model to address complex 3d geometry when modeling more fields (e.g. velocity field and stress tensor field) in the future.
> or physics-informed upgrades of U-Net such as e.g.
>
> - Wandel et al., Teaching the incompressible Navier–Stokes equations to fast neural surrogate models in three dimensions -> an example of physics-informed fluid simulation based on physics-informed U-Net
The work [2] uses a physics-informed loss with U-Net for 3D fluid simulation. The researchers consider channel flow in three geometries: box, ball, and cylinder, in a resolution of 128x64x64 (0.5M) voxels. The Reynolds numbers studied range from 0.64 to 800. They also test the model on the shapes of a fish and three boxes. The model employs finite-difference schemes (eq 6-9) to compute the physics-informed loss. The model has a great performance on these 3D simulations.
In contrast, in our work, we consider a higher Reynolds number regime ranging from 4.35 × 10^5 to 6.82 × 10^6 generated on 7.2M mesh points. We think this scenario is more realistic and intriguing. The physics-informed setting is a promising direction we aim to explore further. However, the finite-difference scheme in [2] introduces truncation errors, which can be problematic in turbulent regimes. Thus, we plan to explore physics-informed neural operators that compute the exact derivatives from the Fourier space. Further, the neural operators are discretization-invariant, meaning they can be training with lower data resolution and fine-tune with the physics-informed loss at higher resolutions, which is an efficient way to combine data and physics.
> or the splitting of the solver into region-wise optimization such as
>
> - Balu et al., Distributed Multigrid Neural Solvers on Megavoxel Domain
The work [3] innovatively proposes the use of multigrid training for neural networks. Similar to the multigrid iteration, the methods explore V, W, and F cycles. It is shown that the deep learning model enjoys a significant speedup. In numerical analysis, multigrid methods are introduced to aid the slow convergence of low-frequency errors, given that the standard Jacobi iteration converges faster for high-frequency errors. On the other hand, data-driven neural network models appear to converge faster on low-frequency modes, which possess higher energy. It would be intriguing to conduct a spectral analysis and study the convergence of frequency.
[1] Brandstetter, Johannes, et al. "Clifford neural layers for PDE modeling." arXiv preprint arXiv:2209.04934 (2022).
[2] Wandel, Nils, Michael Weinmann, and Reinhard Klein. "Teaching the incompressible Navier–Stokes equations to fast neural surrogate models in three dimensions." Physics of Fluids 33.4 (2021).
[3] Balu, Aditya, et al. "Distributed multigrid neural solvers on megavoxel domains." Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2021.