Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?

Summary

This paper addresses the challenge of large-scale dataset distillation, specifically focusing on reducing the storage requirements for auxiliary soft labels in ImageNet-level condensation. The authors propose Label Pruning for Large-scale Distillation (LPLD), which aims to achieve state-of-the-art performance while significantly reducing the storage needed for soft labels. The main contributions include: 1. Identifying that high within-class similarity in condensed datasets necessitates large-scale soft labels. 2. Introducing class-wise supervision during image synthesis to increase within-class diversity. 3. Demonstrating that simple random pruning of soft labels is effective when data diversity is improved. 4. Achieving SOTA performance with significantly reduced label storage, validated across various networks and datasets.

Strengths

1. The paper addresses an often overlooked problem in dataset distillation: the large storage requirements for auxiliary data. Their class-wise supervision approach offers a novel solution to this issue. 2. The analysis is thorough, using feature cosine similarity and MMD to demonstrate increased diversity in synthetic data. Experiments across Tiny-ImageNet, ImageNet-1K, and ImageNet-21K show the method's effectiveness on various datasets. 3. The paper is well-organized and written. The authors clearly explain their motivations, provide detailed methodology, and offer comprehensive analysis of their results. Their figures and tables effectively illustrate key points. 4. By significantly reducing storage needs (40x compression of soft labels) while maintaining or improving performance, this work makes dataset distillation more practical for large-scale applications.

Weaknesses

Limited theoretical analysis: While the paper provides comprehensive empirical evidence, a more rigorous theoretical analysis could strengthen it. I know this is a common challenge in dataset distillation research. However, the detailed empirical study presented here lays valuable groundwork for potential future theoretical analyses in this direction, particularly regarding the relationship between class-wise supervision and increased diversity.

Questions

1. How sensitive is the method to the choice of hyperparameters, particularly in the label pruning process? Is there a way to automatically determine the optimal pruning rate? 2. Have you explored the potential of combining your class-wise supervision approach with other dataset distillation techniques? Could this lead to further improvements? 3. How does the performance of your method scale with the number of classes? Is there a point where the benefits of class-wise supervision diminish? 4. Have you investigated the impact of your method on the training time of the distilled dataset compared to previous approaches? 5. In Table 6(c), why the performance on Swin-V2-Tiny is the worst even if the architecture has the largest size (28.4M)?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes. Limitations are provided in Appendix E.5.

Summary

The paper focuses on reducing the size of soft-label storage in large-scale dataset condensation. The authors discussed why the labels are large-scaled and then proposed to prune the labels by increasing the diversity of synthetic images. Extensive experiments are conducted to validate the effectiveness of the proposed method.

Strengths

1. The paper tackles the problem of large storage space consumed by soft-labels, and discover that this is attributed to the lack of image diversity within classes. 2. Extensive experiments demonstrate the effectiveness of the proposed method, achieving comparable or even better performance than previous methods with reduced label storage.

Weaknesses

The motivation of the work is well-grounded. I don’t observe major drawbacks but do have some minor concerns towards its technical contributions. 1. The proposed method seems to be a class-wise version of SRe^2L, with a soft label-reuse mechanism during training. Thus the technical contributions seem to be weak. 2. How is Equation (8) used in the proposed method? Or is it just a theoretical result? 3. I have doubts on the computational cost (memory and time) of the proposed method (not storage), doing class-wise optimization may increase the distillation time, especially on ImageNet. 4. How is the random label pruning done? E.g. Is the same number of soft-labels removed for each epoch? 5. Proposition 2 is quite confusing, is “higher MMD” a typo?

Questions

Please see Weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes, the limitations are addressed.

Summary

This paper discovers that the conventional method generates images with high similarity. To solve this, the authors introduce class-wise supervision during the image-synthesizing process by batching the samples within classes. Thanks to the increase in diversity, the soft labels can be pruned to reduce the storage size. Extensive experiments are performed on ImageNet with different compression ratios of labels to validate the effectiveness of the method.

Strengths

1. Improving the diversity in condensed images is crucial for obtaining high performance. 2. Label pruning is necessary to reduce the storage size for soft labels.

Weaknesses

1. There are confusing sentences. The authors mention that ”The high similarity of images within the same class requires extensive data augmentation to provide different supervision” and then “To address this issue, we propose Label Pruning for Large-scale Distillation”. How does label pruning address the issue? I see that increasing the diversity of synthetic samples does improve the performance and label pruning is proposed to reduce the storage size. These are not highly related. 2. The authors should compare the performance of the proposed Label pruning with other compression methods (e.g., Marginal Smoothing/Re-Norm with Top-K, using different K when targeting at 10x/20x…) proposed in FKD. Without comparing these methods, it is unclear whether the proposed approach is better than previous pruning methods. FKD: A Fast Knowledge Distillation Framework for Visual Recognition, Zhiqiang Shen et al

Questions

See weaknesses

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

After rebuttal

Thanks for providing the detailed rebuttal. All of my concerns have been properly addressed. In addition, I have also read the other reviews’ comments and corresponding rebuttals. These efforts are appreciated. After careful consideration, I believe this is a high-quality paper that merits acceptance. The proposed soft label compression technique is valuable and contributes significantly to the field. I will adjust my score accordingly once the discussions between other reviewers and the authors have concluded.

Thank you for the response

Thank you for the detailed response and my concerns are addressed. I now vote for acceptance of this paper. However, I strongly recommend the authors to include the discussions on time cost and detailed pruning process into the main paper/appendix. These will be insightful for the community.