Summary
This paper investigates the learning theory of empirical risk minimization (ERM) with feature learning. Under the setting where the optimal finite-sample feature is selected by minimizing the empirical risk over a class of features, the authors show that ERM with feature learning implies convergence of the excess risk under certain assumptions. Moreover, other statistical properties, such as asymptotic normality, are also derived.
Strengths
The statistical theory of this paper regarding feature learning is solid, and the results of the main theorems (Theorem 3 and Theorem 4) seem to be correct, even though I have not had time to check the detailed proofs. In fact, the Glivenko-Cantelli assumptions on the empirical processes and the finite moment assumptions are widely used in learning theory, and the rate $1/n\delta \approx O(1/\sqrt{n})$ seems to be correct.
Weaknesses
1. The title "On the Efficiency of ERM in Feature Learning" seems misleading. Initially, it suggests that ERM should aid feature learning. However, after reading the paper, it appears that the authors are discussing the statistical performance of ERM in the context of feature learning, without clearly demonstrating how ERM aids in feature learning.
2. The authors have reviewed many statistical theory papers. However, in terms of feature learning or representation learning, they didn't conduct a thorough literature review. Specifically, for the final-layer feature, recent studies show that the last-layer feature will converge to an Equiangular Tight Frame (optimal feature for classification problems), which is known as neural collapse, as proposed by Papyan et al. in their paper "Prevalence of neural collapse during the terminal phase of deep learning training." There are some subsequent studies that show that the optimal feature is indeed the minimizer of a regularized ERM. These studies include but are not limited to "Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training" (Fang et al., 2021); "A Geometric Analysis of Neural Collapse with Unconstrained Features" (Zhu et al., 2021); and "Neural Collapse in Multi-label Learning with Pick-all-label Loss" (Li et al., 2024). For different loss functions, the authors may refer to "Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path" by Han et al. Regarding learning theory, the sample complexity under neural collapse is investigated by Wang et al. (2024) in their work "Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learning."
3. Some of the claims are confusing and may be over-claimed. Specifically, in the Conclusion, the authors claim that their theory might be used to explain the double descent phenomenon and generalization of deep learning with label noise in Zhang et al., 2021. However, the feature learning setting in this paper is far from being extended to the deep learning setting. Indeed, their assumptions here, such as the moment assumption, should hold for all feature maps in the hypothesis class, which is not verified for deep neural networks. However, existing feature or representation learning theory such as the Neural Collapse theory can partially explain the phenomenon in Zhang et al., such as the overfitted model can still generalize.
4. As the theory is not as enlightening as the authors claimed in their paper since their setting is not practical, this should be regarded as a purely theoretical paper. As a theoretical paper, there should be some room for improvement, such as verifying the assumptions for all feature maps (such as the moment assumptions in Theorem 4) for a certain class. Moreover, as a purely statistical theory paper, a lower bound showing that the obtained finite-sample rate is optimal is necessary for a high-quality publication. It would also be better for the authors to emphasize more technical difficulties compared to ERM without feature learning. In fact, under the Glivenko-Cantelli assumptions and the finite sample assumptions, both the asymptotic theory (Theorem 3) and the non-asymptotic theory (Theorem 4) look like a simple extension of Theorem 1 and Theorem 2 without feature learning, while Theorem 1 and Theorem 2 are not novel in learning theory.