Reply to questions and discussions
Thank you for your further comments and suggestions.
Regarding A1, pervasive label noise has been present in commonly utilized datasets, with validation studies revealing higher (label) error rates in CIFAR-10 compared to MNIST~[2]. Furthermore, upon closer examination of the double descent plot for MNIST trained without label noise, a noticeable alteration in the shape of the test error curve is observed at the interpolation threshold — a discernible rise. We contend that it is not the case that double descent occurs on CIFAR-10 but not on MNIST; rather, the distinction lies in the level of the interpolation peak, and this difference is due to the different label error rates.
Speaking of A3, the trend in development is enough to make the conclusions and validate our hypothesis. Drawing a comparison between the actual test accuracy and prediction accuracy (expressed as a percentage ${\rm P}$) is meaningless when label noise is absent, as the context for a k-NN comparison becomes non-existent. Our objective is to discern the correlation between the enhanced double descent phenomenon and the model's interpolation strategy concerning introducing label noise when explicit label noise is introduced.
For A5/A6, the better generalization ability of over-parameterized models has been documented through empirical studies showcasing the occurrence of the double descent phenomenon. The primary objective of this study is to unravel the underlying mechanisms driving this phenomenon, particularly exploring its association with label noise in the form of noisy labeled data. We demonstrated that over-parameterized models, when contrasted with under-parameterized ones, exhibit a more precise probability of predicting correctly in the vicinity of mislabeled data points within the training set; While the exact training point are still predicted with the incorrect label by these fully-interpolate models.
We posit that this phenomenon has not been previously documented and could serve to elucidate the internal mechanism of the double descent phenomenon.
Addressing your last inquiry, it is indeed a valid question whether we should compare the noisy labeled data to its counterparts in the training set or the test set. We contend that this choice does not markedly impact the validity of our hypothesis because the neighbors only serve as a convenient means of characterizing local structures in the high-dimensional space. As supporting evidence, we conducted empirical studies in both scenarios (initial submission on the training set; and revision on the test set), and the observations consistently align. Nevertheless, the prediction accuracy on the training set typically exceeds that on the testing set, often reaching full interpolation, and how interpolating networks interpolate the training data is surely more interesting. We have reverted to the initial submission version, incorporating experiment results comparing the training set, and modified the methodology description.
[2] Northcutt, C.G., Athalye, A. and Mueller, J., 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749.