Thanks for your reply. We understand that you are dissatisfied with the dependence of our approximation bounds on the input dimension, which makes them subject to the curse of dimensionality, and we agree that it is important to study alternative function classes for which neural networks can avoid the curse. We strongly disagree, however, with the conclusion that this makes our results uninteresting, or not worth publishing.
In the following, we try one final time to make our point.
1. As we already pointed out in an earlier response, for the $C^k$ function class that we consider, the curse of dimensionality is an inescapable fact of life and cannot be avoided. This holds in a worst-case sense, but quite likely also in an "average case" sense. Indeed, the paper "Phase Transitions in Rate Distortion Theory and Deep Learning" by Grohs, Klotz, and Voigtlaender shows this optimality in an average sense in a slightly, but closely related setting. We are strongly convinced that this result also extends to our setting. Verifying this, however, is outside the scope of this paper.
2. In the real-valued setting, there are several highly influential (well published and highly cited) works that are subject to the same limitations as our result. As selected examples we mention the following:
- "Neural networks for optimal approximation of smooth and analytic functions" by Mhaskar
- "Error bounds for approximations with deep ReLU networks" by Yarotsky
- "Optimal approximation of piecewise smooth functions using deep ReLU neural networks" by Petersen and Voigtlaender
- "Optimal approximation of continuous functions by very deep ReLU networks" by Yarotsky
- "The phase diagram of approximation rates for deep neural networks" by Yarotsky and Zhevnerchuk
- "Deep network approximation characterized by number of neurons" by Shen, Yang, and Zhang.
This underlines that such results are of high interest in the community.
3. Our results are not strictly limited to the class of $C^k$ functions. As an important auxiliary result (which might be of independent interest), we show that CVNNs can well approximate algebraic polynomials. There are natural and widely studied classes of functions that can be very well approximated by polynomials; for instance this holds for certain classes of holomorphic functions; see e.g. Example 2 in the paper "Approximation of smooth functionals using deep ReLU networks" by Song, Liu, Fan, and Zhou. For instance, for this class, our results on the approximation of polynomials using CVNNs would imply that using CVNNs with $N$ neurons, one can obtain an approximation error bound of $C \cdot \rho^{- N^{1/(2n)} / 5}$, where $C > 0$ and $\rho > 1$ only depend on the size of the polyellipse on which the considered functions are holomorphic, but not on the dimension. This bound is still subject to the curse of dimensionality, but much less than our bound for $C^k$ functions. This shows that our results and proof techniques can be useful to tackle the question of alternative function classes for which the curse can be avoided. We emphasize that there are many other function classes that can be well approximated by polynomials; for these, our results will thus be helpful.
We will be happy to add a brief discussion of these points to the final version of the paper.