DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation

Summary

The paper presents a simple but novel approach to enhance image diversity in dataset distillation. previous methods face challenges in balancing computational efficiency and diversity in synthetic images. The proposed EarlyLate training scheme addresses these issues by partitioning predefined IPC samples into smaller subtasks and using local optimizations for each subset.

Strengths

1. The EarlyLate training scheme effectively enhances the diversity of synthetic images. This is a very simple but novel approach to increase the diversity of distilled datasets. I believe it can provide some inspiration for future work. 2. The method, or the training scheme reduces the computational load compared to batch-to-global matching methods. 3. The experiments are comprehensive, including performance, cross-architecture generalization, ablation, and application. These experiments verify the method's superiority.

Weaknesses

1. Compared to previous methods, the work in this paper is incremental. 2. The motivation and the advantages of the "Selection Criteria" in the initialization approach are not clear. And I am confused about how to rank, which is presented in Fig 5, could the authors explain it here? 3. There are a lot of hyperparameters involved. How should these hyperparameters be tuned? Are there any principled approaches? 4. I want to know the impact of the initialization method. In the ablation study, only CDA+init is shown. More advanced methods with init and whether EarlyLate uses init are not presented. 5. The performance of other sota methods on MobileNet-v2 is not presented in Table 1. Is the proposed method still better than other sota methods on MobileNet-v2? 6. Training tricks like random crop play a significant role in methods such as SRe2L. I would like to know to what extent the method proposed in this paper relies on such tricks.

Questions

Please refer to weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper proposes an EarlyLate curriculum learner, which distills the easiest samples first and gradually add harder samples. Based on batch-to-global distillation algorithms, the proposed method consistently enhances the distillation performance.

Strengths

- The writing is clear. - The proposed curriculum learning scheduler seems effective, which is also an interesting point to analyze. - Good distillation performance.

Weaknesses

Limited contribution and potential overclaiming: 1. In section 3, the initialization with real samples is common in DD, the data selection is proposed by RDED, and only the training scheduler is proposed in this paper. I suggest that, at least, add some diversity analysis and comparison of the distilled data. 2. Though the paper is titled with "diversity-driven", the method part lacks justification of the relation between "diversity" and the proposed scheduler. It seems that only the real initialization contributes to the diversity.

Questions

1. Equation 7: I assume that the initialization samples for $\{x_0, x_1, ...\}$ are ordered by the same criteria in Line 168-170, which is not clarified. The sample order here is critical since $x_0$ will be trained for more iterations than $x_{Mk−1}$. A comparison between random order and ascending/descending order by patch probability could rationalize the model design. 2. Table 3(c) is the most important experiment to support the motivation of the whole paper. Maybe the authors could rewrite the story in the future version by: finding these observations -> thorough analysis -> proposing the method. The current table is not comprehensive enough, and more analysis is appreciated: 1. Comparison of 1K/1K is non-trivial (baseline). 2. It is interesting that 4K/2K is worse than 2K/1K, which means training longer leads to a performance drop (they have the same two phases, but 4K/2K trains longer in each phase). The same happens to 4K/1K and 2K/500. 3. In Table 5: 4K-iteration is not fair for comparison since original SRe2L only trains for 1K iterations. A similar concern exists in the model design: e.g. if the base algorithm could converge within 500 iterations, DELT becomes trivial since the algorithms consume all training samples in the last 500 iterations (under default setting MI=4K and RI=500). Minors: - Line 66, footnote links to Fig.2 - A figure with training time as the x-axis and training sample number as the y-axis could make the curriculum learning clearer and easier to demonstrate the computation time reduction.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately discussed the limitations.

Summary

Recent advancements in dataset distillation have led to two main approaches: batch-to-batch and batch-to-global matching. While the former excels in small datasets, the latter, though popular for large datasets, faces a diversity challenge due to independent optimization. Authers propose an EarlyLate training scheme that enhances diversity in batch-to-global matching by partitioning IPC samples into subsets and optimizing them locally. Experiments show significant improvements over previous methods.

Strengths

1) The technical approach is solid and robust, demonstrating a high level of technical competence. 2) The performance metrics presented are highly competitive, showcasing the method's effectiveness in comparison to existing benchmarks.

Weaknesses

1) The motivation for the research is unclear, lacking an explicit articulation of the unifying challenges faced by current state-of-the-art works. 2) The resolution of the figures is inadequate, impeding clear interpretation of the results. 3) There is inconsistency in the styling of table borders and captions, with captions for Table 1 and 2 placed in different positions compared to subsequent tables, some above and some below the table. 4) The experimental settings are not uniformly aligned, and efforts should be made to cover all datasets and settings consistently across all experiments to ensure comparability and rigor.

Questions

see weaknesses

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

see weaknesses

Summary

This work studies batch-to-global dataset distillation, optimizing the synthetic dataset by matching the statistical information of the synthetic batches to that of the full real dataset. Previous batch-to-global methods lacked diversity because each batch had the same optimization objective, leading to redundant information being learned across different batches. Based on this, the paper proposes an early-late training method. First, the real data is divided into lowest, medium, or highest probability patches based on a pretrained model, and these patches are sampled to initialize the synthetic dataset. During training, within-class samples are divided into smaller sub-batches, which are gradually concatenated for batch-to-global training.

Strengths

1. Previous batch-to-global methods indeed faced the problem of synthetic datasets receiving the same supervision signal, leading to redundant information being learned. This paper attempts to propose a new solution to this issue. 2. Extensive experiments demonstrate the good efficacy of the proposed method.

Weaknesses

## The main problem of this work is the writing. >*It dedicates too much space to introducing previous work.* The introduction describes previous methods in too much detail, leading to redundancy with the content in the related work section. The related work section also spends too much space summarizing and describing previous methods. >*The technical part is confused* The proposed method appears straightforward, but the authors describe the entire process almost entirely in text, lacking mathematical descriptions and definitions, which makes it somewhat difficult to understand. I suggest the authors dedicate more space to explaining the Concatenation Training and Training Procedure, incorporating some formulas to clearly demonstrate how the training is conducted. >*It is doubtful whether the proposed method can effectively solve the issues present in previous approaches.* Although the synthetic dataset is further divided within classes and different initializations are used, the supervision signal for each sub-batch seems to still be the same global signal as in other batch-to-global methods. This means that each sub-batch is still optimized in the same direction, potentially resulting in redundant information being learned. I suggest that the authors try to provide a more sound explanation.

Questions

The Training Procedure in Section 3 is somewhat difficult to understand. When training the later sub-batches in DELT, are the previous parts frozen, or are they trained together?

Rating

5

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

N/A