Summary
This paper introduces a novel method for dataset distillation (DD): DD through Directed Weight Adjustment (DWA). The authors first identify that the current dataset distillation methods often fail to ensure diversity in the distilled data, which leads to suboptimal performances. To address this issue, they propose a method that uses dynamic and directed weight adjustment techniques to increase the diversity of synthetic data. The propose method DWA introduces three modifications based on SOTA method SRe2L: 1). Decoupled coefficients for batch norm loss 2). Initialize with real images 3). Random perturbation on model weights. Extensive experiments on datasets like CIFAR, Tiny-ImageNet, and ImageNet-1K demonstrate the superior performance of this approach, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense.
Strengths
1. Increasing diversity in distilled datasets is a key challenge posed by existing methods, and this paper addresses it effectively. The authors conducted a detailed analysis showing that batch normalization plays a crucial role in enhancing diversity in dataset distillation. By leveraging this observation, they proposed innovative techniques to incorporate batch normalization more effectively within the distillation process. The result is a significant improvement over state-of-the-art (SOTA) dataset distillation methods.
2. The authors also conduct detailed ablation studies that demonstrate the effectiveness of each component of their proposed method. The contribution of each element is clearly shown, providing a comprehensive understanding of the method's strengths. Additionally, they show that the method's effectiveness is robust across different hyperparameter settings and datasets, which makes the method have wide practical applicability.
Weaknesses
1. In the ablation study, the authors analyzed and justified the design choices for two key components: the decoupled $L_var$ coefficient and the Directed Weight Adjustment (DWA) scheme. Additionally, they proposed using real initialization. However, the study lacks an ablation analysis that separately evaluates the contribution of these three components to the overall performance of DWA. Such an ablation would provide a clearer understanding of the individual and combined effects of the decoupled $L_var$ coefficient, the Directed Weight Adjustment scheme, and real initialization on the method's efficacy.
2. The paper primarily presents results using moderate to large data budgets (10 and 50 IPCs) for most datasets and 100 IPC for ImageNet-1K. Since the notion of “diversity” only becomes significant when each class has multiple prototypes, the proposed method’s benefits are limited to situations when the distillation budget is high. Overall, the proposed method brings sizable performance improvement but only for one SOTA method with specific data budget constraints.
Questions
1. The paper considered using a directed perturbation to increase the diversity of the model. Is it equivalent or similar to using model trained on different random seeds?
2. In section 2, the authors pointed out $\vec{x}$ and $\vec{s}$ are in latent space, mapped by $g_{\theta}$, following the same notation, Eqn. 2 is operating on latent space and the trained classifier $f_{\theta}$. Is the random weight perturbation method proposed in Section 3.2 only on the classifier part then (Alg 1 only refers to $f_{\theta}$ or the entire network $g$ is perturbed?
3. In section 3.2, how is min/max taken over $L$, if I am not mistaken, $L$ is a gradient of the loss function on $\theta$?
Limitations
Yes, the authors adequately addressed the limitations of their work.