The second part of Rebuttal
(We apologize for reaching the word limit earlier, and we provide the second half of the rebuttal here.)
### Summary of the Theoretical Significance of the Paper
*"The bulk of the paper is really just setting...... the experiments within the main text."*
Thank you for your suggestion. We will summarize the theoretical significance of the paper within the background of NTK theory.
- Background of our work: For a long time, NTK theory has played an important role in the study of network. Under NTK theory, a series of works tried to explain the remarkable generalization ability of network [lai2023generalization], [li2023statistical], [lai2023generalization2], which applied mirror initialization and derived the minimax optimality of network.
- Impact of standard initialization: However, we wonder how the actual results would be if the standard initialization are considered. Under standard initialization, our result shows that the generalization ability of network under NTK theory is actually quite poor. It is a surprising result and prompts us to think further.
- Significance of our work: On one hand, this tells us that the generalization ability of network is not only influenced by the smoothness of the goal function but also by the way of initialization. Thus, we have to carefully take the way of initialization into consideration. On the other hand, we can consider whether the inconsistency between the theory and reality is due to the overly strong assumptions of NTK theory (e.g., the width of network is sufficient large). In the future, it may be worthwhile to explore how NNK changes during training under finite width to advance the existing NTK theory.
- Experiment: In the experiment of Section 5.1, we chose a single-layer network with width $m =20n$ to ensure the width is sufficiently large. At each $n$, we conducted 10 experiments with a learning rate of 0.6, and each training was run for 10n epochs to ensure sufficient time. We will add these details to the main text. Thank you for your suggestion.
### The Problem of Fig 1
*"The synthetic experiments (Fig 1) only cover ..... theory could have trouble with."*
Thanks for pointing this out and we appreciate your correction. Indeed, the range of our $n$ is not large enough. In kernel theory, it is common for the generalization error to decrease polynomially with respect to $n$ (e.g., Fig 1 in [li2024saturation]). In Fig 1, we aim to demonstrate a similar phenomenon. However, in fact, it indeed may encounter the problem of double-descent as you mentioned. To ensure the convergence of NTK, the width of the network $m$ needs to increase with $n$, but it will result in a high computational cost. The reasons above are why we didn't increase $n$. Thank you for correction again.
### Response to Questions
1. The notation $\sum a_i \lambda_i^{s/2}e_i(x)$ is mainly because $\lbrace\lambda_i^{s/2}e_i(x)\rbrace$ forms an ONB of the interpolation space. Thank you for pointing this out. We will explain this in more detail.
2. Under mirror initialization, for $f^*\in[\mathcal{H}^{NT}]^s$ ($s \geq 1$), the generalization error of network is $n^{-\frac{s(d+1)}{s(d+1)+d}}$, as shown in [li2023statistical]. We appreciate your reminder and we will add this point to the discussion of Theorem 4.3 for the convenience of comparison by readers.
3. Thank you for suggestion on the notations in the experimental part. We estimated the smoothness of real datasets with respect to the NTK of single-layer fully connected network. We will add this point in our paper.
4. Regarding the estimation of decay rate in the log-log plot, there may be some miscommunication. As a function of $n$, e.g., in Fig 1, we denote the experimentally measured generalization error as $e_i(n)$, where $n$ is the sample size and $i$ represents the $i$-th experiment. Then $e_i(n)$ is approximately of the form $e_i(n) \propto n^{-\alpha}Z_{n,i}$, where $Z_{n,i}$ denotes a random variable with values around 1, and $\alpha$ is the decay rate to be estimated. In this case, a linear fit after a log transformation may not be a bad choice. We found that Clauset et al. studied estimating $a$ through i.i.d. samples $x_{i} \sim p(x) (\propto x^{-a})$, which is slightly different from our problem. We guess there may be some miscommunication here.
5. Thanks for your careful reading again. We apologize for our annoying typos. We have tried our best to read through the paper and will make several modifications in the new revision.
### Impact of Network Architecture on Our Results
Our results indeed focus on fully connected network. Thanks for pointing this out. For other architectures, like ResNet or CNN, if we know the properties of the corresponding NTK, RFK, and initial output function, it is also reasonable to apply our framework. This requires considerable preliminary work, but in the future, we are willing to explore whether our results hold under different network settings. Again, thank you for highlighting this point.