Summary
Authors provide theoretical results on sample efficiency for learning holomorphic neural operators $\mathcal{X}\rightarrow\mathcal{Y}$ (results are available for both Hilbert and Banach spaces) with a neural network of a particular structure. The structure of the network is encoder $\mathcal{X}\rightarrow\mathbb{R}^{d_1}$, decoder $\mathcal{Y}\rightarrow\mathbb{R}^{d_2}$ and DNN $\mathbb{R}^{d_1}\rightarrow \mathbb{R}^{d_2}$, with both encoder and decoder considered to be given and DNN trained on $l_2$ loss in a usual supervised manner.
Besides the upper bound on approximation error, authors also provide lower bound and conclude that active learning (deliberate collection of data used to train neural operator) is not justified, since non-adaptive training is already optimal up to log terms.
Authors illustrate their theoretical results (relative test error vs number of samples for selected DNN configurations) with several numerical experiments, including diffusion equation in $D=2$, Navier-Stokes-Brinkman equation in $D=2$ and Boussinesq equation in $D=3$.
Strengths
Operator learning is a relatively recent DL setup, so novel theoretical results are of considerable interest. Authors managed to significantly improve previously known bounds and show that DNN can obtain near-optimal performance (in terms of the number of samples needed) in non-adaptive iid settings.
The contribution is primarily theoretical, but formal statements are well explained, supplemented with discussions and examples which significantly simplifies understanding of the results.
Weaknesses
Overall, in my opinion, the research is of excellent quality, so I can not point to any major weaknesses.
The two main issues that can be relevant are: (i) the requirements for the encoder and decoder are not sufficiently discussed, and I find restricting the assumption that they are already available, and (ii) the significance of the numerical confirmation is not apparent.
Questions
1. After line 102 there is a definition of Bochner and Pettis $L^p$ norms. Presumably, the later expression is a definition of $|||F|||$ which is not apparent.
2. I want to clarify several statements on the encoder and decoder.
1. In (1.3) authors introduce encoder $\mathcal{E}$ and decoder $\mathcal{D}$, and later approximate encoder and decoder $\widetilde{\mathcal{E}}$, $\widetilde{\mathcal{D}}$ after line 114. What is the difference between those?
2. In particular, in the discussion (lines 145, 146) of assumption A.IV (line 131) that requires both $\mathcal{E}$ and $\mathcal{D}$ to be linear and bounded, authors claim that $\widetilde{\mathcal{E}}$ is not required to be linear. It is not clear to me how this agrees with the fact that $\mathcal{E}_{\mathcal{X}}$ is defined in A.III as a composition of approximate encoder and decoder. Can the authors please clarify that?
3. According to (1.3) encoder and decoder are defined for the approximate operator $\hat{F}$. This means, as I understand, they are not rigidly fixed by $F$ since presumably $F$ can be approximated with many different encoders and decoders. Approximate encoders and decoders are not formally defined, so it is not clear what they approximate. Does it all mean that authors consider not a continuous operator learning problem but suppose that discretization (of the space of parameters and $\mathcal{Y}$) is already performed ($\mathcal{E}$ and $\mathcal{D}$ are selected) and $\widetilde{\mathcal{E}}$ and $\widetilde{\mathcal{D}}$ approximate $\mathcal{E}$ and $\mathcal{D}$?
4. Why there are $\widetilde{\mathcal{E}}\_{\mathcal{X}}$ and $\widetilde{\mathcal{D}}\_{\mathcal{X}}$ but not the approximate encoder and decoder for $\mathcal{Y}$?
3. Several questions on Theorem 3.1:
1. Why $a_{\mathcal{Y}}$ is called approximation error? What is the meaning of this term (it looks like some kind of volume)? In the ideal setting $\mathcal{D}\_{\mathcal{Y}}\circ\mathcal{E}\_{\mathcal{Y}} = I$, so we have a perfect approximation that has nonzero approximation error.
2. Since $\lim_{m\rightarrow\infty}m \big/ L(m) = \infty$ many terms in the upper bound will diverge unless the encoder and decoder are of sufficient quality. More specifically, the encoder and decoder should improve their accuracy as $\sim m^{-1/2}$ for u.b. to be useful in case of $2$ norm and as $\sim m^{-1}$ in case of Chebyshev norm. Can the authors please discuss this consequence of their result in more detail?
3. Can the authors also explain the consequence of the optimization error term? It seems that $\tau$ that appears in it is to some extent beyond control and completely independent of $m$.
4. Questions about experiments:
1. In experiments authors strive to confirm theoretical scaling $m^{-1}$. Since the encoder and decoder are perfect, and noise is absent, the only two sources of error are approximation error and optimization error. In my opinion, one can clearly see the signs of saturation for larger $m$. This is also clear from operator learning literature that relative error rarely fell below $10^{-3}$. Can the authors please comment on that? Is it because optimization error prevails for large $m$?
2. Dimensions $d=4,8$ in Figures 1, 2, 3 and in the text (e.g., lines 296, 297: "PDE in $d=4$ and $d=8$ dimensions") are slightly misleading because they refer to the dimension of the parameter space and can be confused with the number of spatial dimensions.
3. All results are given for perfect encoder and decoder. Can the authors share some thoughts on whether it is possible to test their upper bound with an imperfect decoder and encoder? What are those experiments might be? Is it possible to create a controllable setting to test terms in ub that contain encoder and decoder errors?