Summary
**SUMMARY AFTER REBUTTAL**: as described below, most of my comments were addressed and the authors made a significant job in including new results. I have increase my score during the rebuttal phase and I strongly vote for acceptance.
---
Neural operators are neural networks that are trained to approximate a function-to-function mapping. This paper proposes an algorithm to perform few-shot learning of these models, where at inference time the network is given a set of examples of the operator, and an input function as the test point.
The specific algorithm they propose is built on reproducing kernel Banach spaces, where for a given prediction, the kernel and the dictionary on which it is evaluated is computed by a recursive embedding of the few-shot examples. This is inspired by the transformer architecture. When examples are continuous, they are discretized with an FFT.
They analyze a series of classical benchmarks including modeling PDEs, showing the method to perform well.
Strengths
As far as I know, the idea of performing few-shot learning of neural operators is novel. These networks are used extensively in physical modelling, so the paper can have a sizable impact there.
The paper is also relatively clear (with a few exceptions, see below), especially if one considers the complexity of some underlying ideas. The connection between their method and the transformer is also interesting, but from what I understand this is extended from Wright and Gonzalez (2021).
The experiments are varied and cover a wide range of use cases.
Weaknesses
1) I have found the initial discussion on the difference between transduction and inference a bit misleading. According to Vapnik, 2006 (which they cite), almost anything which is used today in ML / deep learning is inductive. Transductive methods are only those that can make predictions on a given set of test points but *cannot* operate outside those. For example, SVM is inductive in general, except for some variants such as the transductive SVMs discussed in Collobert et al., 2006. In fact, what they are calling "transductive" is what is typically called "instance-based" in ML (kNN, SVM). Their setup is a standard few-shot setup extended to operators. This also leads to some strange sentences, such as "inductive neural learning with gradient descent is compute-intensive" referred to neural networks; SVM training is also notoriously intensive (and it can also be done via gradient descent).
2) There are some "standard" methods for performing few-shot learning in the literature (e.g., MAML, prototype networks, ...). Many of these methods start from a standard neural network and adapt it to a few-shot scenario. Architectures for performing operator learning are known, and it is not clear from reading the paper why the few-shot learning methods are not immediately extensible to this setup (e.g., why can't we do MAML on a standard neural operator?), and why we need instead to resort to a more complex formulation in terms of kernels. To clarify: I think the algorithm shown here is interesting, but it's a bit hard to motivate it by reading the paper itself.
3) The computational complexity of the method is not discussed, in particular the need to execute multiple FFTs for the few-shot examples (Section "Discretization"). This also connects to recent literature on performing FFTs in an efficient way on GPUs (e.g., FlashConv). Also for testing, if I understand correctly the authors are mostly comparing their method (which is only a forward pass) with a full training of its competitors, which is a bit unfair. No baselines for few-shot learning are tested. For example, they state that their setup is "Similar to (Pathak et al., 2022)", but they do not compare.
Questions
- I think rephrasing the entire "transductive" discussion in terms of few-shot learning could significantly improve the paper. At the least, objectively incorrect sentences should be amended.
- Discussing better training and test complexity is important.
- I think the paper can improve readability by providing a practical example of (5)-(6) in the more familiar case of finite-dimensional input-output spaces.
- There has been a limited amount of works that have explored using neural networks to perform Bayesian posterior inference on-the-fly on a large family of distributions (e.g., Prior-Data Fitted Networks (PFNs)). I would be curious to see a discussion on the connection with this work. On the related work part, there is also some relevant literature on recursive (recurrent) kernel evaluation which is not mentioned.
- There is a small typo on page 3: point evalutation.
- Will the code be released?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors are clearly discussing the limitations of their few-shot setup. However, they are not discussing the computational complexity of the method, which I think would be the biggest limitation in scaling this up to large setups. A few sentences on this would be appreciated.