Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra

We thank the reviewer for their thorough evaluation of our manuscript and their constructive comments. We appreciate the positive feedback on the clarity of our writing and the soundness of our results. We address the reviewer's concerns below. i) Regarding the statement on line 191, we agree with the reviewer that the preactivations are indeed Gaussian because they are linear combinations of Gaussian inputs. The Gaussian equivalence principle is only necessary when discussing the post-activations, not the preactivations. We have corrected this in the manuscript by removing the reference to the Gaussian equivalence principle in that context. ii) Concerning line 417, we acknowledge the missing factor of $ l $ in the exponent of the eigenvalues in the expression for the variance of $T_{nm}^{(l)} $. We have corrected this mistake in the revised manuscript. To compute the scaling of the variance with $ N $, we note that $ \lambda_l = \lambda_+ / l^{1+ \beta} $, and the convergent sum $\sum_{l=1}^N 1/l^{2 l (1 + \beta)} $ is of order one, not of order $N $. Therefore, the variance scales as $ \lambda_+^{2 l} / N^2 $, and the expectation value scales as $ \lambda_+^{l} / N $. This implies that the ratio of the standard deviation to the expectation value remains of order one, indicating that higher-order overlaps do not self-average. This lack of self-averaging leads to the presence of $ M $ distinct plateaus in the dynamics of quantities like $ R_{in}^{(l)} $, as observed in our results. Questions: i) Regarding the discrepancy in line 190, we thank the reviewer for pointing this out. We have corrected the expression for the covariances now in l.189 such that they are consistent with the definition of preactivations (which were correct). ii) For line 223, we appreciate the reviewer's attention to detail. The reviewer is correct that the expression refers to the minimal polynomial, not the characteristic polynomial. We have updated the terminology in the text to ensure clarity. Minor comment: i) We agree that plotting the predicted slope for the power law and indicating the associated window of validity would enhance the illustration of our theory. We have updated Figure 2 (right panel) to include the predicted slope and have extended the analysis in Figure 9 in the appendix to test additional configurations. This also allowed us to improve the upper bound, making it tighter and more consistent with the observed behavior. ii) We concur that demonstrating the scaling of the escape time is valuable. In the left panel of Figure 5, we now show the linear dependence of the plateau length on $M^2 $, and in the right panel, we illustrate the $ 1/L $ dependence. This provides experimental support for our theoretical findings regarding the scaling of the escape time $ \tau_{\text{esc}} $. iii) Regarding the intuition behind why the length of the plateau decreases with $ L $ and increases with $ M $, we offer the following explanation. The network output scales with $ 1/\sqrt{M} $ due to the averaging effect of the $ M $ hidden units. Consequently, the generalization error and the gradient scale with $ 1/M $. Learning $M $ features is inherently more challenging than learning a single feature, which contributes an additional factor of $ M $ to the plateau length, leading to the $ M^2$ dependence. As for the $ 1/L $ dependence, normalizing the average covariance to one means that the largest eigenvalue of the input covariance matrix scales with $ L $. Directions associated with larger eigenvalues are learned more rapidly, causing the departure from the plateau to occur sooner and shortening the plateau length as $ L $ increases. We hope that these clarifications and additions address the reviewer's concerns. We have updated the manuscript accordingly and are grateful for the opportunity to improve our work based on the insightful feedback.

We thank the reviewer for their thorough evaluation of our manuscript and for their constructive comments. The reviewer is mainly concerned about the generality of the findings in our work and asks about possible generalizations. i) Regarding the case when $K \neq M $: The scenario where $ K > M $ is interesting because, even for isotropic input patterns, the generalization error obeys a scaling law $ \epsilon_g \propto \alpha^{-2} $ in the asymptotic regime of large $ \alpha $(see Phys. Rev. E L052302 (2022)). We have added an analysis of this case to the appendix, based on a solution to differential equations for the dynamics of the student vector. We find that the asymptotic power law is independent of $K $; that is, the power law determined by the data covariance matrix is not modified due to the intrinsic power-law learning dynamics of the model. This is presumably because the power-law exponent $\beta / (\beta + 1) $ is always smaller than the intrinsic power-law exponent. The results are provided in Appendix D. On the other hand, the case $ K < M $ would be a generalization of the situation where the student cannot fully learn the teacher weights due to a lack of parameters. For every $ K < M $, the generalization error would reach an asymptotic plateau value, and we expect a scaling $\epsilon_g \propto K^{-\gamma} $. Since this additional scaling law does not affect the generality of our results, we have not performed additional analysis for this rebuttal. ii) Concerning the scenario where the student weights are not identical but equal to the teacher weights, we expect the scaling law to be unchanged. When the student weights are not identical and differ from the teacher weights, we anticipate that perfect learning is no longer possible. Hence, the generalization error reaches an asymptotic plateau beyond which there is no power-law scaling anymore. Perhaps the most interesting generalization is when the student weights in the second layer are trained. In this situation, one finds that the exponent for learning does not change for the error function activation; it remains unchanged at $\beta / (\beta + 1) $. However, for ReLU activation, the learning exponent does improve, becoming larger than $ \beta / (\beta + 1) $. We have added a solution to modified differential equations describing this situation to the appendix. When training the second layer, the resulting fixed points for the error function activation correspond to the weight configuration of the soft committee machine. In contrast, the dynamics for the ReLU activation lead to a new fixed point that deviates from the soft committee machine configuration. Future research could focus on understanding the mechanisms behind improving the learning exponent for ReLU activation and why this improvement is absent for the error function activation. We provide the results in Appendix F. iii) Regarding the case where the covariance matrix is only partially degenerate: If we start from a situation with $ L = N/2 $ and a two-fold degeneracy, which is then lifted, we expect a smooth crossover to the case $ L = N $, in which no degeneracy occurs. We anticipate similar crossovers when higher-order degeneracies are lifted. iv) As for other activation functions, we have added solutions to differential equations describing the case $ K = M = 2 $ with a ReLU activation function. The results are provided in Appendix E. Additionally, we performed simulation experiments for $ K = M = 2 $ using various activation functions (see Figure 27), where a student is trained on the outputs of a teacher using synthetic input data. We find that the asymptotic power law remains unchanged compared to the error function activation. Regarding the questions raised: 1) The dependence of the generalization error on $ N_l $ is intended to mimic the scaling with the size of the network, analogous to the study by Maloney et al. (2022) [1]. In the main manuscript, we examine a simplified setup with a diagonal covariance matrix, so each trainable component of the student vector corresponds to learning a direction in input space with a specific variance. In this sense, each entry of the student vector directly corresponds to an eigenvalue of the covariance matrix. We have revised this section to enhance clarity (cf. line 312). The behavior we observe in the simplified model does not change when we relax the condition of a diagonal covariance matrix; please see the new section in Appendix B.3.1. 2) We have included the widths of the plateaus in the figure to demonstrate the scaling with $ M $ and $ L$. Specifically, we have added a new figure (Figure 5, left panel) to illustrate this quantitatively. $\mathbf{Continuation\ of\ discussion\ in\ the\ next\ comment}$

We thank the reviewer for their thorough evaluation of our manuscript and for their constructive comments. We are pleased to address the questions and concerns raised. 1. Most of our results are not self-averaging with respect to a random teacher vector $\boldsymbol{B}$. The generalization error depends on the order parameter $T^{(1)} = \boldsymbol{B}^\mathrm{T} \boldsymbol{\Sigma} \boldsymbol{B}$ (inline equation above Eq.~(5)), which is not self-averaging. This lack of self-averaging arises from the power-law distribution of eigenvalues in $\boldsymbol{\Sigma}$ implying that the order parameter is dominated by a relatively small number of components of the teacher vector. Therefore, we have explicitly averaged over the teacher vectors in the results presented in our manuscript. 2. Regarding the CIFAR-5M plot in Figure 3, we used labels generated by a teacher network. The question about scaling laws with the true labels is intriguing. However, the classification performance of committee machines on CIFAR-5M is not sufficient to enter a scaling regime. We believe that training a more powerful architecture, such as a ResNet, on CIFAR-5M with the true labels would be necessary to observe scaling. We expect that the scaling in this case could exhibit a larger exponent than predicted for our setup, due to the training of multiple layers (see the new plot in Figure 25 of Appendix F in our revised manuscript). In general, scaling exponents depend on the network architectures. For CIFAR-10, Rosenfeld et al. [1] used wide residual networks (WRN) and found a scaling exponent $\beta \approx 0.5$with the model size, whereas Sharma and Kaplan [2] found $\beta \approx 0.23$ for a simple CNN. To provide additional information on how the plot in Figure 3 was obtained: We used the parametrization from Eq. (80) to produce the results shown in the right panel. We tested our prediction for the generalization error from Eq. (79) on a student network trained on CIFAR-5M images using approximately $10^6$ input examples (see Nakkiran et al., 2021 for details on the dataset). We used only the first channel of the images, resulting in a total input dimension of $N = 1024$ after flattening. To approximate the true covariance matrix $\boldsymbol{\Sigma}$, we numerically estimated the feature-feature covariance matrix based on input examples from the training dataset. During training, we updated only the first $N_l$ entries of $\tilde{\boldsymbol{J}}$, resetting the remaining entries to their initial values after each iteration. Based on the spectra of the feature-feature covariance matrix depicted in Figure 14, we estimated $\beta \approx 0.3$ and used this spectrum to evaluate Eq. (80). 3. In response to the request about plotting Figure 6 on a log-log scale, we have replotted the left panel accordingly. We would like to point out that the inset of this plot was already on a log-log scale in the original version of our manuscript, allowing us to observe the dependence of the scaling exponent on $\beta$. Additionally, in response to another reviewer's request, we have added Figures 22 and 24 to the appendix, displaying the solution of our differential equations on a log-log scale for both error function and ReLU activation functions. 4. Regarding whether the committee machine learns features in the linear network case, for a linear activation function, it cannot learn features. As argued at the beginning of Section 4.1, the teacher output is fully determined by the average teacher vector, i.e., the student does not have access to the outputs of individual teacher units and cannot learn the features encoded by them. 5. Concerning the sensitivity of the solutions in the committee machine to the initialization of the student weights, as described by Eq. (12), the variance with which the student is randomly initialized determines the length of the symmetric plateau. According to Eq. (15), the length of this plateau depends on the teacher's initialization, not on the initialization of the student. Therefore, we expect the dependence on the student's initialization to be weaker than the dependence on the teacher's initialization and to have little effect on the asymptotic power law. In our numerical experiments, we have simultaneously changed the initialization of both the teacher and the student when performing averages. Our differential equations show that the plateau length depends on the initial conditions, specifically on the $Q^{(l)}$, which are not self-averaging. However, the fluctuations in the plateau length are less pronounced compared to those arising from different initializations of the teacher. We hope these clarifications and additions adequately address the reviewer's concerns. We have revised the manuscript accordingly and appreciate the opportunity to refine our work based on the valuable feedback.

Dear Reviewer fBBP, We sincerely appreciate your valuable time and insightful comments on our work. As the deadline for the Author-Reviewer discussion draws near, we kindly request your feedback on whether our responses have effectively addressed your concerns. Thank you once again for your contributions, and we hope you have a wonderful day! Best regards, The Authors