Summary
This paper presents Kernel Neural Operators (KNOs), a new method for operator learning that employs deep, kernel-based integral operators with quadrature to approximate function spaces. By utilizing parameterized, closed-form kernels that are finitely smooth and compactly supported, KNOs can reduce the number of trainable parameters compared to neural operators like the Fourier Neural Operator (FNO). Additionally, KNOs are well-suited for handling irregular domains and non-uniform data.
Strengths
- Developed a novel neural operator modeled as a kernel integral operator.
- Proposed a new method of parameterizing kernel integral operators with radial Wendland kernels.
- This approach minimizes the number of trainable parameters in KNO, making it effective with fewer parameters.
- Capable of processing inputs on arbitrary grids.
Weaknesses
- The authors conducted multiple experiments, yet the complexity of these experiments may be limited. For example, the solution operators for 1D PDEs are relatively straightforward, and in the Navier-Stokes experiment, the low Reynolds number leads to a very smooth vorticity field. Given that most of KNO's parameters come from the MLPs within the integral layers, it remains uncertain how much the kernel integral layers contribute or if simple MLPs could approximate the solution operators just as effectively.
- I believe the comparison between KNO and the baselines was not entirely fair. Rather than running their own baseline experiments, the authors relied on results reported in other papers. For many cases, the specific architectures used are unclear, which raises concerns about the transparency and applicability of these comparisons. This issue is especially evident in the FNO model used for the Darcy (triangular-notch) experiment, where only 8 Fourier modes were applied, suggesting suboptimal model selection. Additionally, the FNO model has demonstrated an overall relative L2 error of around 0.8% for an autoregressive task that maps the first 10 time steps to the last 40, step-by-step, using the updated implementation released over two years ago. However, the authors reported a single-step error of 1.81% mapping the first 10 steps to the 20th step for FNO; this is possible, but no details and their own implementations are provided, which raises doubts about the fairness of these comparisons, as essential details to verify the results are missing. For greater transparency, the authors should have conducted their own baseline training and model selection. The authors should include experimental details in greater extend.
- No convolution-based baselines were run, despite their demonstrated effectiveness in learning PDEs (such as U-Net, [1], [2]).
- The experiments conducted do not substantiate the claim that KNO performs well on irregular domains. The authors should compare KNO with additional baselines designed for irregular and more complex domains to better demonstrate its effectiveness (such as [3]).
- The novelty and contribution of this work may be somewhat limited, as the overall framework largely mirrors that of FNO, with the primary differences lying in how the kernel and kernel integral are managed. However, the authors do not provide enough discussions and evidence, either theoretical or empirical to convey their claims.
**Overall this paper presents an interesting idea. If the concerns and questions regarding the experiments can be answered satisfiedly, I would be happy to increase my score.**
[1] Towards Multi-spatiotemporal-scale Generalized PDE Modeling; Jayesh K. Gupta, Johannes Brandstetter; 2022
[2] Convolutional Neural Operators for robust and accurate learning of PDEs; Bogdan Raonić, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, Emmanuel de Bézenac; 2023
[3] Operator Learning with Neural Fields: Tackling PDEs on General Geometries; Louis Serrano, Lise Le Boudec, Armand Kassaï Koupaï, Thomas X Wang, Yuan Yin, Jean-Noël Vittaut, Patrick Gallinari; 2024
Questions
- Why didn’t the authors conduct their own experiments? It is important given that the reported results, e.g. for NS, are from a paper published in 2021 (on Arxiv 2021), updates to the FNO architecture since then have led to notable improvements.
- Can the proposed architecture be implemented via standard convolutional layers if the input domain is sampled on a regular grid?
- The experiments were not done on the original FNO data and setup, and wondered if the KNO can outperform the FNO on that setup? Clearly, the NS example is not the original autogressive setup from the first 10 time steps to the last 40. The Darcy flow results for FNO is also worse than other what FNO can achieve (implementation updated after the release of the original paper, but the update is from 2 years ago).
- "In section 2.1, integral operators: FNOs use an implicitly-defined, dense, matrix-valued kernel that couples all channels of the architecture. In contrast, the KNO enforces sparsity on this global matrix by utilizing a diagonal matrix-valued kernel."
Can you clarify this? As I understand it, your approach in each kernel integral layer does not use in_channels $\times$ out_channels kernels. Instead, you employ a number of kernels equal to the number of input feature maps/functions, performing integration independently for each channel without interaction across channels. While this indeed reduces the parameter count significantly, it may compromise performance, particularly in comparison to FNO. Moreover, channel interaction is standard in CNNs. Could you provide an explanation or rigorous justification for why this method would not lead to accuracy degradation?