Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy

Summary

This paper studies data pruning in noisy label setting. Built on re-labeling models, it proposes Prune4ReL that finds a subset to maximize the re-labeling accuracy. In particular, it introduces neighborhood confidence as the criteria for selection, as well as a greedy algorithm to select the subset. Evaluation show its superiority over prior data pruning methods.

Strengths

The target setting is interesting and well motivated. The proposed neighborhood confidence and Prune4ReL is sound with theoretical guarantee. In addition, the evaluations are comprehensive and experimental results look promising.

Weaknesses

I don't see any major weakness, but I do have some comments: 1. The proposed data pruning method does require model training as many sample selection methods for robust learning do; I think these methods should also be considered as baselines and compared, even though they are not specifically designed for data pruning. Currently, the authors only compare SmallLoss, which is quite outdated. 2. There are some related work in robust learning field that also leverage neighborhood information, but the authors did not include and discuss them, eg, "Learning with Neighbor Consistency for Noisy Labels"

Questions

See weakness above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors clearly stated the limitations in appendix.

Summary

This paper studies the task of data pruning, specifically in the setting of label noise. The authors propose a method to perform data pruning by maximizes the total neighborhood confidence of the training examples (which is equivalent to maximizing the relabeling accuracy). The authors theoretically analyze this particular setting, bounding the error of a model trained on the subset that satisfies the expansion and separation assumptions from prior work. Their resulting bound is inversely proportional to the neighborhood confidence. Their empirical results show mixed results in the comparison against existing methods; namely, in Table 1, the results are slightly better than existing baselines on a small number of tasks but predominantly match the performance of existing methods. In table 2, it also seems that the existing method Forget and kCenter seem to select similar or fewer noisy examples than the proposed method (except in the case of CIFAR-100N with 0.2 or 0.4).

Strengths

The authors present a theorem that shows that the error of a model trained on a subset of data is inversely related to the neighborhood confidence. They propose a method that maximizes the neighborhood confidence, which in turn minimizes this bound. The authors show in Figure 3 that the test accuracy with SOP increases when using a subset produced by the proposed method. However, it is unclear why this is necessarily the case, since there are similar or fewer noisy examples selected by the Forget baseline method. The authors provide ablation studies that study the effect of neighborhood threshold $\tau$ and with different confidence metrics.

Weaknesses

The bolding strategies in Table 1 are a bit misleading; in many scenarios, a baseline achieves the same performance as the proposed method (and sometimes achieves a smaller variance), but the proposed method is still listed in bold. In fact, the reported method achieves the same performance across many of the tasks. Similarly, the results in Table 2 show that the Forget baseline seems to select subsets with a smaller ratio of noisy examples when compared to the proposed method. However, in subsection 4.3, the authors claim that this method selects a high ratio of noisy examples.

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

The paper proposes a novel data pruning algorithm, Prune4ReL, that maximizes the neighborhood confidence of the entire training examples, which is proportional to the likelihood of correct re-labeling. The paper demonstrates the effectiveness of Prune4ReL on four noisy datasets, where it outperforms baselines by a large margin.

Strengths

The writing of this article is very clear and easy to follow up. Moreover, the methodology in this paper is also reasonable with necessary theoretical analysis. I enjoy this work very much! My only concern about this work is the experimental part.

Weaknesses

1. Recent works like [1,2,3] are missing in those selected baselines. 2. Lack of results on ImageNet-1K, which is the most convincing part for us. [1] Moderate: Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. ICLR-2023 [2] SSP: “Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS-2022 [3] CCS: Coverage-centric Coreset Selection for High Pruning Rates. ICLR-2023

Questions

In addition, the current dataset-pruning experiment mainly focuses on image classification, but this kind of task is too simple to reflect the real needs. For example, for CIFAR or even ImageNet, we don't need dataset-pruning at all, and the cost of training on these datasets is very affordable for almost all scientific teams. I think the real application scenario of dataset-pruning should be the most popular tasks such as large language model (LLM) training and multi-modal training. So, what do you think is the main difficulty of using dataset-pruning for these tasks? How is these tasks different from image classification? At present, the performance of dataset-pruning algorithm in image classification tasks is generally not very good, for example, the performance degrades very seriously under high pruning-ratio Settings. Moreover, all existing pruning algorithms cannot outperform the ramdom selection baseline with a significant margin. However, in the field of LLM, as a contrast, some manually filtered datasets [1,2] can achieve a data set size reduction of tens of times, and performance remains unchanged or even significantly improved. So, can we say that manual filtering is still far better than algorithmic automated filtering? Or, what in the world is causing such a big gap. [1]. Textbooks Are All You Need. Arxiv. [2306.11644]  [2]. LIMA: Less Is More for Alignment. Arxiv. [2305.11206]

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to Weaknesses.

Summary

The paper proposes Prune4ReL, which prunes a noisy training dataset such that the performance of a Re-labeling trained downstream model is maximized. Unlike previous work, the paper targets pruning a *noisy* dataset and explicitly considers the learning algorithm of the downstream model. The proposed method is specifically tailored to Re-labeling, automatically cleaning the noisy training dataset. The utility function in Prune4ReL is inspired by the theory proposed in [1] and gracefully incorporated into the context of data pruning. The experimental results show that most prior work suffers under noisy datasets, while Prune4ReL remains robust to the noise in the training dataset, resulting in substantial improvements over baselines. [1] Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data, ICLR2021

Strengths

The strength of the paper include - Clear presentation and easy-to-follow writing - The proposed method is theoretically-inspired and, maybe more importantly, easy to implement - The evaluation, together with the analysis, is convincing. - Include the analysis on why and how the baseline fail The paper is well-organized, and the method and evaluation are solid.

Weaknesses

The paper conducts a complete study on the proposed Prune4ReL, and the following weakness is relatively minor. - Prune4ReL outperforms the baselines by a large margin, but the gap between PruneReL and uniform sampling is small. - Some notations are not clear. - Def. 3.1: the $x$ is not in the pixel space. Instead, it's in the embedding space. The author should state the dimension of $x$ at the beginning - in Thm 3.4, it should be $\mathcal{S} \subseteq \tilde{D}$

Questions

Experiment - Uniform sampling is usually the second-best baseline. It performs especially well in CIFAR-10N Random/Worst and Clothing-1M. Can the authors elaborate more on this? - Do the authors believe the gap between uniform sampling and Prune4ReL is small due to the greedy sampling? Or what are the other hypothesis? - The derivation from the reduced neighbourhood confidence (Eq. 3) to the empirical reduced neighbour confidence (Eq.4) is confusing. What motivates using the cosine distance to perform a weighted sum? Is this motivated by importance sampling? - WebVision and Clothing-1M are crawled from the web, but Prune4ReL seems to work better in Clothing-1M. Can the authors elaborate on this further? - It seems that kCenter and GraNd are missing in Figure 3b. The authors have to find a better way to visualize this plot (Figure 3b). Try to zoom in a little bit. - Section 4.5 is like the motivation of the paper. Consider re-order them.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

As mentioned in the #weakness, there are some minor limitations. - Empirical results: The proposed method still requires some work on improving over uniform sampling baseline, and the authors did not justify the strong uniform baseline.

Post rebuttal

I read the author's rebuttal and would keep my score. In addition, I think this work is also related but missed in discussion/comparison: Resolving Training Biases via Influence-based Data Relabeling.

Reviewer response

Thanks for your response. I appreciate the changes for the bolding in Table 1 and the clarification for the distinction between noisy and self-correctable datapoints, as well as the additional results on synthetic flips for ImageNet-1k. I'm happy to increase my score to a 5 but am not wholly convinced by the empirical results.

Reviewer response

Great work! The authors addressed all my concerns and please include the discussion of the uniform sampling baseline in the final version. I will keep my score.

Thanks!

Congrats! I am very satisfied with your reply! This is a good article and very enlightening! I have a habit of giving conservative marks in the first stage review! I hope it will not affect you badly.