Official Comment by Authors (2/2)
**(I) Assumption 3.2 and scaling with $\delta$.**
> (scaling with $\delta$) Theorems 3.4. ... on $\delta$.
> the result holds with probability degrading depending on the validity of this approximation
First, these are correct.
> (in "scaling with $\delta$") for some confidence levels, there are large numbers of $m$ verifying the assumptions of the paper, is not more interesting than an infinite $m$ assumption.
> (in "Assumption 3.2") I do not see why the finite $m$ regime satisfied by Assumption 3.2. is **any more** interesting than infinite $m$.
As the reviewer pointed out, for some confidence levels, there are infinitely many values of $m$ that satisfy the conditions, naturally including infinite width. However, this does not imply that we must choose infinite width. We can select the smallest $m$ that satisfies the conditions.
Moreover, we do not claim that the finite width regime is more *interesting* than infinite width. The finite width is discussed in this paper because it is not theoretically necessary to consider infinite width. There may be some misunderstanding between us and the reviewer regarding the term "interesting." If our response does not address the reviewer's concerns, we would appreciate further clarification.
> There is nothing in the practice of ML that would make me think Assumption 3.2. is more reasonable than the infinite $m$ assumption.
We have interpreted this concern as follows:
*In practice, the experimenter does not know in advance how large $T$ (training time) needs to be for perturbation learning to succeed. If the experimenter continues training until perturbation learning succeeds, $T$ could become very large. Therefore, the experimenter may need to select a very large $m$ in advance, which is essentially equivalent to assuming infinite width.*
This concern is valid when $T$ is variable during training and unknown in advance, and the experimenter is free to choose an arbitrarily large value.
However, it should be noted that our theory does not claim how the training evolves over time (i.e., $T$; variable). It characterizes the model trained with a designated setup (including the training time $T$; constant). In other words, we fix $T$ before training, as is typical in actual network training. For example, it is natural to set $T$ in advance to 100 epochs for MNIST. The experimenter selects a constant $T$ (although it does not always lead to the success of learning) and, accordingly, a finite $m$, which is more reasonable than an infinite width assumption.
**(II) Assumptions in Theorems 3.4., 3.5.**
> there are two competing conditions on $T_f$ and $T_g$ ...
The reviewer might consider that $T$ is substantially constrained to $c < T < C$ for some $c,C$, which prevents the satisfaction of the functional margin conditions. We should note that the assumption $T = \Theta(1)$ is introduced only for notational simplicity of the functional margin conditions. Essentially, no assumption on $T$ without $T > 0$ is required. Simply speaking, this assumption is introduced to derive $O(T + (1/\sqrt{d})) = O(T)$. Although not realistic, if $T = \Theta(1/d)$, then $O(T + (1/\sqrt{d})) = O(1/\sqrt{d})$. Without any assumptions, we would need to write unnecessarily intricate conditions, which we believe should not be included in the main text. We will revise any misleading assumptions.
> there are remainder terms ... which are not discussed.
These time-independent terms are discussed in Lines 179--181.
**(III) Network parameters change during the training.**
The reviewer's understanding is generally correct. The reviewer seems view "feature learning" as the extraction of higher-level features that are potentially latent within the data. In other words, they interpret features not as raw data vectors or their parts but as more complex combinations of these elements. According to the reviewer, kernel methods and our framework (cf. Eq. (22) and (23)) perform "feature alignment" with raw data rather than "feature learning" as a higher-level feature extraction.
I partially agree with this assertion. The difference in intepretation between us and the reviewer seems to be in the scope of what is referred to as "features." In addition to these high-level features, we also consider the data vectors themselves as features and regard their simple extraction as a form of "learning." For example, in the binary classification of vertical and horizontal lines or in MNIST, the raw data itself would serve as features. However, we will revise any misleading terms that may lead to misinterpretation. Furthermore, we will explicitly state in the limitations section that we prove the feature hypothesis and perturbation learning from primitive feature perspective, but do not fully explain them with higher-level features.
Note that prior work shares the same limitation, and the extraction of higher-level features might require relaxing constraints related to depth rather than width.