Summary
This paper studies the benign overfitting (or not) of two-layer ReLU neural networks under the hinge loss. Under a new noisy (almost orthogonal) data setting without separable data setting,
- Theorem 3.1 shows that benign overfitting exists occurs the signal-level parameter $\gamma$ is small in $\Theta(1/n)$;
- catastrophic overfitting (the generalization error is bounded below by a constant) occurs when $\gamma$ is small enough at the order of $n^{-2/3}$.
- Instead, if $\gamma$ is large, no overfitting occurs, i.e., training error cannot be zero but generalizes well.
Strengths
- provide solid analysis of generalization guarantees of two-layer ReLU neural networks under a new data setting
- The results on benign overfitting, catastrophic overfitting, and no overfitting heavily depend on the choice of the signal-level parameter $\gamma$
Weaknesses
- The motivation of this paper is to fix the benign overfitting problem under the hinge loss. Since the analysis of previous work on the logistic loss is invalid, the authors focus on a new data generation setting and new proof framework. However, the motivation and significance of this research appear somewhat limited, particularly when considering the hinge loss. It would be beneficial to emphasize the differentiation between the results obtained with the logistic loss and the hinge loss, highlighting the theoretical advantages of employing the hinge loss. Additionally, exploring potential implicit biases related to benign overfitting and engaging in further discussions, particularly in comparison to the work conducted by Shamir (COLT2022) on the squared hinge loss, would strengthen the paper's arguments.
- The problem setting and data assumption are a bit unrealistic (also as suggested by the authors). It’s ok for me but there are some assumptions missing, e.g., in the benign overfitting part, $n - k > 2*10^6$ in Lemma B.4 is needed. I think this should be clearly mentioned in the main text.
Minior:
- The early phase in line 224 is undefined.
Questions
- I understand the data setting with $d \geq n^2$ for the orthogonal condition, which is widely used in the benign overfitting literature for two-layer neural networks. But currently there exists some work that has fixed this issue:
Frei et al. Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. COLT2023.
So I kindly ask the authors to think about whether their assumption can be relaxed or not.
- Since I’m an emergency reviewer, I don’t have enough time to check the proof. I high level read the proof framework, and find that line 236-237 is they key proof idea:”every training point activates one neuron after the last training update”. Could you please detail this?
The generalization performance (whether benign overfitting occurs) only depends on the signal-level parameter $\gamma$ in the data generation process. However, intuitively, the difference on generalization guarantees is based on the changes of model complexity. So how to understand this issue?
- Continuing the above question, we can find that, if $\gamma$ is small (enough) at the $O(1/n)$ order, we have the benign overfitting. However, it means that the signal level $v$ is quite small and the noise level $n_i$ is quite large. That means, the model can fit such large random noise? If $\gamma$ is large in Theorem 3.8, the model cannot fix the noise to achieve zero training loss if the noise level decreases. When checking these two results, I find that there exists some contradictions here? Maybe I’m missing something.
- The width $m$ is not required to be over-parameterization? More discussions are needed.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.