Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.https://github.com/AngusDujw/Diversity-Driven-Synthesis.

Paper

Similar papers

Peer review

Reviewer LurV6/10 · confidence 4/52024-07-07

Summary

This paper proposes a dynamic adjustment mechanism to enhance the variation component in batch normalization loss so that increase the diversity of synthetic images. By adjusting the weights, the synthesized images achieve sufficient diversity to cover the widespread distribution of the real dataset.

Strengths

a. Originality: The author innovatively analyzes the different components of the loss function. This work differs from previous contributions by proposing a directed weight adjustment algorithm to increase the diversity of synthetic images. Related work is adequately cited. b. Quality: The submission is technically sound. The claims are well supported by theoretical analysis and experimental results. It is a complete piece of work. The authors are careful and honest in evaluating both the strengths (increased diversity and minimal computational expense) and weaknesses (unexpected noise) of their work. c. Clarity: The submission is clearly written and well organized. d. Significance: The results are important. The submission addresses the diversity problem more effectively than previous work. Most results advance the state of the art with a large margin, providing a unique theoretical approach that is also demonstrated in experiments.

Weaknesses

c. Clarity: A few equations, such as (14), need more explanation. The current text does not adequately inform the reader on how to reproduce the results, as there is insufficient connection to the implementation details. d. Significance: The results on CIFAR10 using ConvNet are significantly worse than the current state of the art. Additionally, the authors did not mention whether or how the method can be integrated as a plug-in.

Questions

1. In equation 14, can the authors explain why it is <=0? In line 176, is the first item clearly greater than the second item because equation 14 is <=0, or is the causal relation inverse, with the first item being greater than the second item concluding that equation 14 is <=0?. 2. In Table 1, Can the author explain why the results of ConvNet on CIFAR10 are obviously worse than state of the art? 3. Why there is no results for ipc=1 in all result tables.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitation about the unexpected noise.

Reviewer e5no6/10 · confidence 4/52024-07-07

Summary

This paper proposes a new method to enhance dataset distillation by improving the diversity of synthesized datasets. The authors introduce a dynamic and directed weight adjustment technique to maximize the representativeness and diversity of synthetic instances. Theoretical and empirical analyses demonstrate that increased diversity enhances the performance of neural networks trained on these synthetic datasets. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, show that the proposed method outperforms existing approaches while incurring minimal computational expense.

Strengths

- The paper is well-written and structured. - The analysis of the decoupled variance component is interesting. - The proposed dynamic adjustment mechanism is novel and can enhance the diversity of the synthesized dataset. - The paper provides a solid theoretical foundation and extensive empirical validation across multiple datasets and architectures. - The findings have important implications for the efficiency of dataset distillation, particularly for large-scale datasets.

Weaknesses

- The effectiveness of the diversity in generated datasets has been discussed in many prior works [1, 2]. - The authors should compare the proposed method with recent SOTA dataset distillation methods that extended from SRe2L [2, 3]. Especially for [3], the relation to it needs to be clarified. - A comparison with recent diffusion-based generative dataset distillation methods [4, 5] is also needed because they are free of optimization. - How long does it typically take to search for the optimal values of $K$ and $\rho$ for ResNet-18 and other different datasets and network architectures? Can the authors provide insights into the computational cost associated with this parameter-tuning process? - Are there any scenarios where the directed weight adjustment might negatively impact performance or diversity? If so, how can these be mitigated? - Have the authors considered some downstream applications to further verify the effectiveness of the proposed method, such as continual learning and privacy protection? [1] Liu, Yanqing, et al. "Dream: Efficient dataset distillation by representative matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [2] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [4] Gu, Jianyang, et al. "Efficient dataset distillation via minimax diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [5] Su, Duo, et al. "D4M: Dataset Distillation via Disentangled Diffusion Model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

See the weaknesses section. I am happy to raise my score if the above concerns can be addressed.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discussed the limitations.

Reviewer NmcN6/10 · confidence 4/52024-07-08

Summary

This paper introduces a random directed perturbation strategy to the teacher models for further strengthening the diversity of the existing sample-wise data synthesis method. Through intuitive empirical and theoretical analysis of the diversity factors, the authors proposed a simple yet overlooked weight adjustment strategy, showing promising results in various vision models and benchmarks including ImageNet. ========== Post rebuttal ========== Thank you for the detailed responses and additional analysis. Having read the responses, my major concerns were addressed. Therefore, I raise the rating.

Strengths

- Intuitive motivational analysis of how the existing methods, SRe2L, lack of diversity, while providing empirical and theoretical backgrounds for the factors that can affect diversity during the synthesis process. - Proposed simple yet overlooked perspectives of existing data synthesis loss via 1) perturbing the teacher model in a random directed way, 2) decoupled variance matching, both of which collaboratively enhances sample diversity. - These simple yet intuitive methods exhibit largely improved performance on various vision models and benchmarks including large-scale ImageNet dataset. - Thorough ablation studies with the decoupled variance coefficient, weight adjustment, and cross-architecture additionally provide an intuition of how the proposed method enhances the actual performance.

Weaknesses

- The proposed method has a heavy reliance on BN statistics matching from CNN-based models. Considering that the recent State-of-the-art models such as ViT are not leveraging BN, the proposed approach might not be directly applicable to these vision transformer-based models. - Failure cases of DWA should be more clarified. For example, In Table 1, when the model and the number of classes in the target dataset is small (ConvNet / CIFAR-10), DWA underperforms compared to the other methods. - Comparison with baseline (SRe2L) in terms of computational overhead is required. Since this paper introduces add-on procedure, weight adjustment, to the baseline, the authors should clarify if the proposed method does not induce significant computation overhead during synthesis. - In Figure 5, the test performance varies depending on the perturbation parameters. Although the authors deterministically set one set of parameters for all the different datasets, it should be clarified if these hyper-parameters are not sensitive and generalizable to all the datasets, to reduce time-consuming trial and error for searching these parameters on the other general datasets.

Questions

- Is the proposed synthesis method applicable to the ViT-based models that leverage LayerNorm rather than BatchNorm layers? Further experiments (directly synthesizing on ViT-based models or cross-architecture experiments by synthesizing with CNN models and testing on ViT-based models) would be required for the generalization of the proposed method to the recent models. - Disregarding the computational overhead, can searching for the optimal perturbation parameters per sample rather than using one set of predefined universal perturbation parameters help improve the performance? - Can you provide comparison of computation overhead with SRe2L on ImageNet dataset?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See above weaknesses and questions.

Authorsrebuttal2024-08-14

Thank you for your insightful and constructive feedback on our submission! We have noted your post-rebuttal response and will incorporate the DeiT results and the computational overhead comparison into our paper, as you suggested.

Reviewer Uwti6/10 · confidence 4/52024-07-09

Summary

This paper introduces a novel method for dataset distillation (DD): DD through Directed Weight Adjustment (DWA). The authors first identify that the current dataset distillation methods often fail to ensure diversity in the distilled data, which leads to suboptimal performances. To address this issue, they propose a method that uses dynamic and directed weight adjustment techniques to increase the diversity of synthetic data. The propose method DWA introduces three modifications based on SOTA method SRe2L: 1). Decoupled coefficients for batch norm loss 2). Initialize with real images 3). Random perturbation on model weights. Extensive experiments on datasets like CIFAR, Tiny-ImageNet, and ImageNet-1K demonstrate the superior performance of this approach, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense.

Strengths

1. Increasing diversity in distilled datasets is a key challenge posed by existing methods, and this paper addresses it effectively. The authors conducted a detailed analysis showing that batch normalization plays a crucial role in enhancing diversity in dataset distillation. By leveraging this observation, they proposed innovative techniques to incorporate batch normalization more effectively within the distillation process. The result is a significant improvement over state-of-the-art (SOTA) dataset distillation methods. 2. The authors also conduct detailed ablation studies that demonstrate the effectiveness of each component of their proposed method. The contribution of each element is clearly shown, providing a comprehensive understanding of the method's strengths. Additionally, they show that the method's effectiveness is robust across different hyperparameter settings and datasets, which makes the method have wide practical applicability.

Weaknesses

1. In the ablation study, the authors analyzed and justified the design choices for two key components: the decoupled $L_var$ coefficient and the Directed Weight Adjustment (DWA) scheme. Additionally, they proposed using real initialization. However, the study lacks an ablation analysis that separately evaluates the contribution of these three components to the overall performance of DWA. Such an ablation would provide a clearer understanding of the individual and combined effects of the decoupled $L_var$ coefficient, the Directed Weight Adjustment scheme, and real initialization on the method's efficacy. 2. The paper primarily presents results using moderate to large data budgets (10 and 50 IPCs) for most datasets and 100 IPC for ImageNet-1K. Since the notion of “diversity” only becomes significant when each class has multiple prototypes, the proposed method’s benefits are limited to situations when the distillation budget is high. Overall, the proposed method brings sizable performance improvement but only for one SOTA method with specific data budget constraints.

Questions

1. The paper considered using a directed perturbation to increase the diversity of the model. Is it equivalent or similar to using model trained on different random seeds? 2. In section 2, the authors pointed out $\vec{x}$ and $\vec{s}$ are in latent space, mapped by $g_{\theta}$, following the same notation, Eqn. 2 is operating on latent space and the trained classifier $f_{\theta}$. Is the random weight perturbation method proposed in Section 3.2 only on the classifier part then (Alg 1 only refers to $f_{\theta}$ or the entire network $g$ is perturbed? 3. In section 3.2, how is min/max taken over $L$, if I am not mistaken, $L$ is a gradient of the loss function on $\theta$?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations of their work.

Reviewer e5no2024-08-09

Thank you for your prompt response, your clarifications have resolved many of my concerns. Regarding A2, I would also like to know the performance comparison when the IPC is set to 10.

Authorsrebuttal2024-08-09

Due to length constraints, these details were not included in the initial response. The comparison with SOTA methods under ipc=10 is summarized below. Benefiting from the more powerful supervisions, such as multi-patch stitching in RDED [2] and the use of large pre-trained diffusion models in Minimax [4], these methods outperform ours in the ipc=10 setting. Despite the limitations imposed by a restricted distillation budget, which hinders the diversity of our synthesis results, our method still performs significantly better than other SOTA methods. **ImageNet-1K** - ResNet-18 | ipc | DWA (Ours) | SRe2L | RDED [2] | G-VBSM[3] | Minimax[4] | D4M[5] | |:---:|:----------:|:-----:|:--------:|:---------:|:----------:|:------:| | 10 | 37.9 | 21.3 | 42.0 | 31.4 | **44.3** | 27.9 | - ResNet-101 | ipc | DWA (Ours) | SRe2L | RDED[2] | G-VBSM[3] | D4M[5] | |:---:|:----------:|:-----:|:-------:|:---------:|:------:| | 10 | 46.9 | 30.9 | **48.3** | 38.2 | 34.2 | Besides the suggested related works, we also review three other recent methods: DATM [6] (a lossless distillation method), CDA [7] (an extension of SRe2L), and D3M [8] (a Diffusion-based method). The comaprsion showcases the cutting-edge performance of our method. **Cifar-100** ConvNet |ipc|DWA (Ours)|DATM [6]| |:---:|:----------:|:-------:| |10|**47.6**|47.2| |50|**59.0**|55.0| **ImageNet-1K** ResNet-18 | ipc | DWA (Ours) | CDA [7] | D3M [8] | |:---:|:----------:|:-------:|:-------:| | 10 | **37.9** | - | 23.57 | | 50 | **55.2** | 53.5 | 32.23 | | 100 | **59.2** | 58.0 | - | [6] Guo Z, Wang K, Cazenavette G, et al. "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching." International Conference on Learning Representations. 2023 [7] Yin, Zeyuan, and Zhiqiang Shen. "Dataset distillation in large data era." 2023. [8] Abbasi A, Shahbazi A, Pirsiavash H, et al. "One Category One Prompt: Dataset Distillation using Diffusion Models". arXiv preprint arXiv:2403.07142, 2024.

Authorsrebuttal2024-08-09

Thank you for your time and constructive comments! The comparison mentioned above will be included in our camera-ready version.

Reviewer LurV2024-08-09

Thank you for your thoroughly explanation. I will maintain the current score.

Authorsrebuttal2024-08-09

Thank you for your time. We are so glad the responses addressed your concerns!

Reviewer Uwti2024-08-11

Thank you!

Thank you for the detailed explanation and all the additional experiments! Addressing diversity in dataset distillation and aiming for lossless distillation is an important topic. I would like to keep my score of acceptance!

Authorsrebuttal2024-08-11

Thank you for your time! Your reviews have helped us improve our work, and we’re glad that the response meets your expectations!

Program Chairsdecision2024-09-25

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC