You Only Condense Once: Two Rules for Pruning Condensed Datasets

Summary

This paper proposes a combination method for dataset pruning and dataset condensation. Both methods could reduce the size of training datasets. Experiments show that the proposed method achieves state-of-the-art performance on several datasets regardless of networks. The proposed rules are simple and effective, which might be commonly used in future works.

Strengths

- This paper studies an interesting yet rarely studied problem, which is important for on-device scenarios. In addition, the authors have comprehensively analyze the challenges of this problem and the drawback of existing methods. - This paper is well-written and easy to follow. I generally enjoy reading the story, motivation, method and experiments of this paper. - Two novel, simple, and effective rules/metrics are introduced to select informative and balanced samples. Importantly, with this two rules, one can carry on various size requirements by one process. - Extensive experiments, including results and visualization, demonstrate the effectiveness and efficiency of the proposed method, and show the superiority over the state-of-the-art methods.

Weaknesses

Generally, I think the paper is strong. However, I have the following questions. - In Table 1, the ImageNet-10 on the DREAM method is not listed. - Some proofs, such as Proof of Theorem 1, can be shortened. - The core idea is first to condense the dataset before pruning the dataset. How about pruning the dataset first and then condensing it? Which way is better and why?

Questions

- From Figure 1, previous methods require 200K *(N-1) epochs for extra condensation. Is there any way to reduce the extra condensation process without using dataset pruning? Dataset Pruning is just like a selection method. - Will the code be open-sourced?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

This paper target at the dataset condensation for on-device scenarios. It proposes two novel rules to prune the condensed dataset. The first rule is to rank the condensed images. The rule believes easy examples are better for pruning condensed dataset. The second rule is to choose balanced samples for different classes. The overall writing is clear. The experiments are sufficient.

Strengths

1. This is the first work to consider pruning the condensed dataset. The novelty is good. 2. The illustration and writing are clear. The motivation is clearly explained in the figures. 3. The paper provides solid proof for the LBPE metric for ranking the condensed images. 4. The balanced construction is reasonable expecially when the dataset size is small. 5. The experiment looks good. On CIFAR-10, CIFAR-100, and ImageNet, the proposed YOCO method surpasses other methods by a large margin. For example, on CIFAR-10, the improvements are from 6.57% to 24.33%. On CIFAR-100, the improvements are from 4.77% to 22.13%. 6. The analysis is interesting. It's interesting to see that the rules are different between the condensed dataset and the full dataset.

Weaknesses

1. The LBPE score and balanced construction. Which rule is more important? 2. If we view the condensed dataset as a full dataset, will the rules change? Is there any threshold for when the rules will change? 3. The LBPE Score is from the Top-K Training Epochs with the Highest Accuracy. Why not use the loss to extract the LBPE score?

Questions

Please see the Weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The limitation part is covered in section 5.

Summary

This paper employs a dataset pruning approach on a condensed dataset. Specifically, the intent is to evaluate and rank the data within this condensed dataset based on their respective significance. This enables users to selectively choose any number of condensed datasets, offering both efficacy and flexibility in training the model.

Strengths

This paper claims that it's the first work to conduct dataset pruning on condensed datasets to fit varying computational constraints. LBPE score is a reasonable metric to measure the importance of data in the condensed dataset.

Weaknesses

1. The main weakness is the evaluation part. This paper compares their method: (1). dataset condensation + dataset pruning with the previous method (2). dataset condensation + random selection (line 202), which is not fair. For example, it's better to show the effectiveness of the proposed method by comparing (1) with IPCF = 10 to IPCT=1 or 2 or 5 with (3): directly generating the dataset condensation with the IPC=1 or 2 or 5. If it can get similar performance, this paper can claim its advantages of training time. 2. It's unclear—the whole pipeline in Fig 1 or the rest of the paper. I can only understand the paper after reading the Appendix. It's more critical than putting the algorithm part in the main body. 3. Proof of Theorem 1 is not solid. 4. The imbalanced part is not thoroughly analyzed. The equal number of each class is a standard setting in the condensed dataset. This paper does not provide new insight from Section 3.3

Questions

Please refer to Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Summary

--- Post-Rebuttal Edit --- After author rebuttals, I have updated my rating from 5 to 6 and confidence from 2 to 3. --- End Post-Rebuttal Edit --- This paper considers a practical problem in dataset condensation/distillation, where edge devices have varying constraints that need to be flexibly met by a data distillation method. The authors propose two rules to prune the condensed datasets in order to save computational costs. The first rule uses logit-based prediction error (LBPE) to identify useful examples, where low LBPE samples are useful for smaller datasets and high LBPE samples are useful for larger datasets. The second rule focuses on the construction of balanced classes by ensuring an equal number of samples in each class via Rademacher complexity and generalization error.

Strengths

- The writing is generally easy to follow and the experiments are thorough. - The proposed approaches are logical and appear useful. The first rule based on LBPE scores allows for flexible treatment based on dataset size and other characteristics; the second rule incorporates Rademacher complexity in a novel way in order to set up a theoretical foundation for the significance of balanced class construction.

Weaknesses

- Perhaps this is because I am unfamiliar with data distillation literature, but the motivation and setting are somewhat unclear to me. There is not a single citation or brief explanation of what is meant by “on-device scenarios.” Even if this is obvious to well-informed readers, I believe that at minimum a citation or definition is needed. - In what scenario would one need to store a training set on a device? Typically when deploying a machine learning model to a hardware-constrained device, the only thing being deployed is the model weights. - Related to the above point, I don’t understand the motivation for the experimental setting, where we are condensing an already-condensed dataset. If one of the central motivations mentioned in the Abstract is to “[eliminate] the need for extra condensation processes,” then I don’t see how this method achieves said goal – we are still ultimately performing two condensation steps! - The mixture of theoretical- and prose-style writing is awkward at times. For example, Theorem 1 on line 122 would be much more appropriately written as simple explanatory text. I don’t understand in what sense this is a theorem or how lines 123-124 constitute a “proof” for this claim. There is no need to present content in a mathematical style when it is not necessary.

Questions

- What is an “on-device scenario?” What is the motivation for storing a training set (rather than, say, the trained model weights) on said device? - What is the motivation for condensing an already-condensed dataset? - How does this method avoid extra condensation steps if all experiments are performed after an initial condensation step with another method? - Why not perform a single condensation step with the proposed method (from full -> desired size)? How does the method perform in this setting?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- See Questions above - There is no clear definition of what is meant by “small” and “large” datasets. A definition or rough empirical guideline would be useful.

Thank you for the clarification

I appreciate the effort put forth in this paper and recognize the novelty in exploring dataset pruning, a direction that I believe remains relatively under-researched. I do have several points of contention and suggestions that I would like to highlight: Q1. The main concern still exists. Referring to Table 1, the paper demonstrates that using IDC directly to produce IPC=10 results in an accuracy of 67.5%. However, when the proposed dataset pruning is employed on a condensed dataset with IPC=10 from IPC=50, the accuracy drops to 60.31%. Such a significant decrease in accuracy challenges the practicality of this pruning method. The primary objective of pruning is to derive a compact model (dataset in this paper) with only a slight compromise on accuracy when compared to a pre-trained model. And this pruned model should outperform training from scratch. While I acknowledge that pruning can speed up the process when leveraging a pre-trained dataset, this significant drop in accuracy remains concerning. My central point here is, I'd prefer comparisons between the Pruned Condensed Dataset and directly obtaining a Small Condensed Dataset with the same IPC. If the latter yields better results without requiring the intermediate step of dataset pruning, then what merits does the pruning method offer? Q2. I do not have any major concerns here. This isn't a critical issue but more of a suggestion for improvement. Q3. The proof provided for Theorem 1 doesn't strike me as formal proof; it seems more like an elucidation. I'd recommend refining this section to meet the standards of rigorous proof. Q4. I acknowledge and commend the analysis of the imbalanced part. My primary issue lies in "The equal number of each class is a standard setting in the condensed dataset." this is not a new insight. In my understanding, the primary aim of dataset pruning is to extract a succinct yet informative condensed dataset. So the goal is to compare with the methods of the condensed dataset. Previous methods of condensed datasets seem to consistently adopt the Balanced construction. For this point, there is not a technical concern but only a novelty concern. In summary, I do acknowledge and commend the contributions of 1, 2, and partial 3 in the paper. However, because this paper is to focus on the first work using dataset pruning on the condensed dataset, authors should emphasize the effectiveness of dataset pruning in this field, such as litter accuracy drop from pre-trained condensed dataset or better than generated condensed dataset directly generated from scratch (IPC, DREAM), instead of focusing on comparing with other unproposed dataset pruning condensed dataset method. Regarding the experimental results, It would be more compelling to demonstrate superior performance when using the Pruned Large Pre-trained Condensed Dataset compared to directly generating the Small Condensed Dataset persist.

Response to rebuttal

Thanks for your effort in rebuttal. My concerns have been well addressed in the response.