Convolutional Neural Operators for robust and accurate learning of PDEs

Summary

The authors propose a UNet-like architecture for learning solution operators of PDEs. The architecture is assembled from building blocks analogous to the classical UNet architecture, but individual components respect a continuous-discrete equivalence. This makes the proposed architecture a representation equivalent neural operator in the sense of a recent paper (Bartolucci et al., 2023). The authors show that under some continuity assumptions the proposed method can learn the solution operators to a variety of PDEs. They also compare the in- and out-of-distribution performance of their method to a variety of baselines for a suite of different types of PDEs.

Strengths

- There is a large body of work on methods and architectures for operator learning which this paper builds up on. Having architectures that respect the continuous-discrete equivalence seems very significant to me. - The presentation is clear and the paper overall well written. - The proposed CNO architecture improves the overall performance for a comprehensive suite of problems and appropriate baselines. - Many evaluations and ablation studies are included in the appendix.

Weaknesses

- As discussed in the related work section, the building blocks that make up the CNO architecture have been introduced in previous papers (Karras et al. 2021, Alias-free generative adversial networks). This slightly weakens the novelty of the paper. - Many definitions are adopted from a relatively recent paper (Bartolucci et al. 2023), which to my knowledge is only available as a preprint at the moment. - The modifications to the UNet architecture introduce additional up-sampling and down-sampling layers, which increase the training time significantly compared to other methods. This needs to be kept in mind. Also, the implementation of CNOs is much more complicated than of e.g. FNOs due to the windowed-sinc filters. - I found the evaluation across resolutions somewhat unclear and partially misleading. The appendix goes into more detail here, but the results in the main paper show a single NS case that resulted from a downsampled solution (hence no high-frequency details exist in the targets). In this case the CNO seems to show a constant error, which is dubious for meaningful real world cases where higher resolutions naturally would exhibit structures that aren't resolved with lower resolutions. I think it will be important for future versions to clarify this, and replace figure 2 with one of the other two cases from the appendix.

Questions

- The practical implementation of the interpolation filters uses windowed-sinc. In theory, doesn't this break the continuous-discrete equivalence? - Regarding "operator" networks, I was wondering why the authors didn't compare to a fully convolutional ResNet (along the lines of Solver-in-the-loop Um'21 or the accelerated CFD from Kochkov'21); that architecture should more naturally extend to different resolutions than a Unet. Can the authors comment on this omission? - Can the authors provide details on the performance, especially how much slower the CNO is compared to the regular Unet?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are only discussed as a future work discussion. I don't think this is sufficient. E.g., a clear discussion of the performance impact is completely missing as far as I can tell.

Summary

In this paper, the authors propose a new operator learning framework called Convolution Neural Operator which although works on discrete space, satisfies the property of Continuous-Discrete Equivalence (CDE) property. They define CNO over a UNet architecture and provide some universal approximation proofs. Further, they come up with a Benchmark of representative PDEs and compare all the existing methods and present superior performance of CNO over other methods.

Strengths

+ Novel idea of CNOs + Theory provided for soundness + Representative Benchmark Dataset and making it public in zenodo is also good. + Very neat and clean code.

Weaknesses

- Some notations are messed up or confusing to follow. e.g. r in line 82. - While the performance of these models is compared... the computation requirements like memory or training time are not compared. I think if we are talking about benchmarking, it is appropriate to give the complete story including the computational requirements. - Some ablation studies on whether using CNOs reduces the architecture/compute requirements compared to just using a CNN or FNO is missing.

Questions

See above in weaknesses*

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Yes. The authors have accurately identified the limitations. There are no negative societal impact for their work.

Summary

The paper under review proposes a CNN architecture as a neural operator for solving PDEs via neural networks. The goal is to preserve the underlying continuous structure while being implemented with discrete convolutional architecture, based on U-Net. The idea is to consider the space bandlimited functions in Sobolev spaces. Defining a convolution layer involves a discrete convolution, upsampling/downsampling that requires sinc interpolation to avoid aliasing, and an activation layer that upsamples, activates and then downsamples to maintain the original band-limitation. The authors prove in supplementary material that their approach can approximate the solution to a PDE with arbitrary accuracy with the CNN architecture based on their neural operator. Experiments are done on solving several PDEs, with experiments favorable to the authors' method against state of the art for most PDEs considered.

Strengths

- The architecture based on CNNs potentially simplifies e.g., FNO operators where operations are done in the Fourier domain without the use of down-sampling - which can potentially be more expensive than the authors' proposed method. - Rigorous mathematical treatment of the interplay between continuous and discrete operators. - Experiments show higher accuracy compared to SOA in PDE solving with NN.

Weaknesses

- No evaluation of efficiency; is there a speed advantage of the proposed method (e.g., compared to FNO)? - Evaluation of parameter sizes is in supplementary and there isn't a clear trend; I would have thought this method would require fewer parameters than e.g., FNO; but in many cases this isn't so. - I would think efficiency in the sense above is a key differentiator compared to SOA, but this isn't the case at least looking at supplementary. - The authors make use of sinc interpolation, which is the ideal interpolator for sequences of infinite length. In practice, you are operating on finite length data, for which the sinc interpolater is not the right one. How applicable is the theory presented to the case of finite length data? - Prop 2.1 talks about a representation equivalent operator and the definition is cited in a reference. Please specify the definition in the paper. - Theory is poorly presented in the paper; everything is in supplementary. At least the key ideas should be in the main paper. - In general many of the key parts that should be in the paper are in supplementary. - I find the authors' proposed idea to set a standard for benchmarking misguided. There is a laundry list of PDEs to approximate in the benchmark, however, I doubt any one architecture would be good for all PDEs. Each PDE has particular properties that one would want to aim to preserve (e.g., conservation laws in some PDEs) that may not be relevant to others. So the implication that one architecture should do well on all PDEs seems mis-guided. I do not think this should be an accepted standard benchmarking.

Questions

- Bandlimited: many solutions to PDEs will have shocks, discontinuities, etc. How relevant is this assumption?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, discussed.

Summary

The paper re-introduces convolution neural networks to the operator learning setting. It uses interpolation including up-sampling and downsampling layer to make sure the convolution layer is well-defined in the function space. The paper also discuss the space of band-limited functions and the trade off between continuous-discrete equivalence and representation power of the infinite dimension function space. The paper proves an approximation theorem for the CNO model and compared it across many partial differential equations.

Strengths

The convolution neural operator (CNO) takes the advantages of the efficient of the conventional CNNs and UNets methods, meanwhile it also satisfies the resolution-invariant and representation-equivalent properties. The work shows an approximation theorem that the CNO models can approximate any continuous solution operator, and the numerical experiments show it has a comparative performance compared to other machine learning methods such as UNet, DeepONet, and FNO.

Weaknesses

I have a few questions and concerns, mainly in the trade-off between the band-limited functions and Lp/Hp space. 1. The paper claim the CNO models can only learn band-limited functions, however, the target PDE system are intrinsically infinitely dimensional system (in Lp/Hp). It means, given a fixed number of parameters (size of the model), as the number of training sampling and the qualify(resolution) of training dataset increase, the truncation error will dominate and the bandlimited class of operator will underperform the band-unlimited models. 2. The band-limited operator is, by definition, linear-reconstruction of the chosen basis, meaning it can only learn the coefficient but unable to learn the basis. The linear-reconstruction are less efficient compared to non-linear reconstruction model, as discussed in [1]. 3. The experiments (Table 1) are designed in a manner that CNO dominates all other methods. It shows the potential of CNO, but the results might be biased. It's certainly reasonable that the band-limited CNO outperform band-unlimited model on super-resolution tasks. However, for in-distribution problems, if the resolution is fixed and the dataset is sufficient, then Unet should be equivalent to CNO, right? And for these non-smooth problems, the band-unlimited could be more expressive than CNO. These cases might not be sufficiently considered in the experiments design. It is great to introduce these new experiments, but it would be better to include as least one of the previous standard benchmark, for example, the Burgers or Darcy equation dataset from [2]. 4. While the perspective is new and interesting, it's hard to say the proposed model is very novel. The architecture is highly similar to the previous UNet model + StyleGAN3 de-aliasing trick. [1] Lanthaler, Samuel, et al. "Nonlinear reconstruction for operator learning of pdes with discontinuities." arXiv preprint arXiv:2210.01074 (2022). [2] Li, Zongyi, et al. "Fourier neural operator for parametric partial differential equations." arXiv preprint arXiv:2010.08895 (2020).

Questions

As discussed in page 4, CNO uses upsampling and downsampling to reduce the aliasing error, but it's also claimed that the activation layer will introduce the aliasing error. This is overcome by "Implicitly assuming that $\bar{w}$ is large enough". Can the author give some examples of the activation such that this is satisfied? Speaking of frequencies expansion, for ReLU or leaky ReLU, $\sigma(x)$ is non-smooth. Even if $\sigma$ is a k-frequency function, $\sigma \circ \sigma$ can be $k^2$-frequency, right? So after several layers it easily goes unbounded. Figure 2 is also a bit surprising. Can CNO achieve constant super-resolution error curve for problem like Navier-Stokes? The FNO error also seems a bit too high. How many modes are used in FNO? If FNO is designed with a reasonably small number of modes and no padding (padding may cause error in resolution), FNO may also get flatten curve.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The author discussed some limitations, but it's also good to discuss the trade-off between using band-limited functions and Lp/Hp space. I think both side have their advantages and disadvantages.

Requesting the reviewer for feedback

Due to the imminent closure of the discussion period, We kindly request the reviewer to provide us with their valuable feedback on our rebuttal and we are at their disposal to answer any further questions in this regard

Requesting the reviewer for feedback

Due to imminent closure of the discussion period, We kindly request the reviewer to provide us with your valuable feedback on our rebuttal and we are at your disposal to answer any further questions in this regard

response to rebuttal

1. Thanks for the additional plots about efficiency, which shows an advantage in the authors' method. 2. I was referring to what you can say theoretically about the finite length case, not empirical. While a good contribution, I thought the presentation could be greatly improved.