MIM4DD: Mutual Information Maximization for Dataset Distillation

Summary

This paper proposes mutual information maximization loss for dataset distillation. Specifically, the authors compute the mutual information between real and synthetic feature distribution in multiple layers by constructing positive and negative pairs. Experiments show that when plugging-in state-of-the-art dataset distillation methods, obvious performance improvements are achieved in multiple datasets.

Strengths

1. The idea of introducing mutual information into dataset distillation is natural and under-explored. It will be an interesting work to the researchers in this field. 2. The paper is well-written and technically solid. Necessary analysis on the learned synthetic data has been provided. 3. Remarkable performance improvements have been achieved when plugging the proposed loss into state-of-the-art distillation methods.

Weaknesses

Incomplete experiment results: a) As a loss function, this paper lacks the experiment results on training with MIM4DD loss independently. b) The ablation study on beta in Table 2 is incomplete, as larger beta has not been tested.

Questions

Typo: 1. Line 135 D_real* -> D_real 2. Left of Figure 2: Synthetic Data -> Real Data

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not addressed.

Summary

Dataset distillation aims to synthesize a small dataset with similar test performance to the original full dataset. This paper argues that previous works neglect information theory considerations, and argues that a well-designed information metric between variables is very important. Therefore, it introduces the concept of mutual information maximization and transforms it into a lower bound optimization problem at the feature map level, which improves the performance of some methods as an add-on module.

Strengths

1. The authors analyze the previous dataset distillation method from the perspective of information theory for the first time; 2. The authors provide a rigorous and reasonable mathematical formula derivation for how to convert the maximization of mutual information into the lower bound constraint of the sample feature map representation;

Weaknesses

1. The reason why the method works is not sufficient. The maximum mutual information describes the degree of correlation between two variables from the perspective of information theory, but does this mean that more classification information can be learned from the synthetic images for the model to improve the quality of distillation? 2. The method is not tested on more popular gradient matching or feature matching based dataset distillation frameworks (e.g. DC, DM, DSA, etc.) as a module. 3. The NCE loss mentioned in line 307 should correspond to the right of Figure 4.

Questions

1. In some datasets, there are bias and noise samples in the same category. For such datasets, is it necessary to use full original dataset information to maximize mutual information? 2. How does the module perform on high-resolution datasets such as TinyImageNet and ImageNet Subset? And if ipc is raised to 50 on cifar100, what is the result?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

see above

Summary

The authors tackle the dataset distillation task — synthesizing a smaller dataset using which one can train models towards comparable test performance to models trained on the full dataset. Unlike the current methods that optimize through heuristic matching between the real and synthetic datasets, the proposed method performs data distribution mutual information maximization. Interestingly, when it comes down to the implementation, the method looks very much like a regular SimCLR-like contrastive learning formulation. However, the authors provide theoretical groundings on how it is related to the mutual information maximization problem.

Strengths

1. It is a fairly simple idea and I am surprised that no one has been doing it — verified by a quick literature search. I suppose one reason is that prior researchers who thought of using mutual information encountered the difficulty of attaining an estimate of the data distribution, which the authors worked around (line 135-163). 2. Illustration in Figure 2 is well-made and very informative. 3. The authors provided theoretical grounding for using the “mundane” contrastive learning formation (Figure 2) by showing its optimization target is closely related to the proposed “Accessible Mutual Information”. In some sense this is an interesting explanation of why SimCLR-like contrastive learning (single instance multi-view, positive/negative samples) works so well. 4. Significant improvement over the baseline (see Figure 3).

Weaknesses

1. At the moment, I am unaware of any work showing the theoretical link between mutual information and SimCLR-like contrastive learning. If there exist prior work on this topic, the novelty of this work could be largely mitigated. 2. It might have been better to explore other more sophisticated datasets and architectures beyond the three-layer ConvNet, though not absolutely necessary.

Questions

1. The authors seem to define dataset distillation in slightly different manners in the abstract compared to in the Preliminaries section. In the former case, they said “dataset distillation aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model” whereas in the latter case they said “the goal of dataset distillation is to synthesize a small training set such that models trained on this synthetic dataset can have comparable performance as models (with the same architecture) trained on the large real set”. The nuance makes the two definitions a bit different — the former is helpful for testing acceleration while the latter is helpful for training acceleration. I would recommend the authors to clarify which definition they decide to go with (or maybe both) and ensure consistency. 2. Is there any reason BPTT+MIM4DD is shown but BPTT is not included as a standalone baseline in Table 1?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Nothing to be noted.

Summary

The paper proposes a new method named: MIM4DD that tries to maximize the mutual information shared between synthetic images and real images during the process of dataset distillation. The paper derives a lower bound of MI and formulates it as a learning objective for optimization. The proposed new loss can be combined with a wide varieties of DD methods to boost performance. ========== I have read the author's responses and they have addressed my concerns by running more experiments and showing more proof. ==========

Strengths

- The paper proposes a new way of boosting the performance of DD methods by measuring the mutual information between synthetic datasets and real datasets - This paper studies the mutual information and formulates it mathematically which is largely ignored by a lot of previous methods. - Through approximation, the paper derives a lower bound of an accessible MI loss and proposes to tackle it with contrastive learning. - The paper is well organized and easy to follow - The comparison is thorough such as Table 1.

Weaknesses

- The foundation of MIM4DD seems incorrect: MI in the targeted data level is equivalent ot MI in the feature maps level. From equation 5, F and G may not be invertible which heavily depend on the activation function. Therefore theorem 1 doesn't apply. - The evaluations results are mostly within the variance of the baseline methods such as MTT, there is no strong evidence that the proposed method works. - Strong limitations are introduced such as 50K negative pairs which could make the method infeasible to apply to larger datasets. - Writing errors such as "to encounter this obstacle"

Questions

- It's hard to interpret the performance boost in Table 1, what are the hyperparameters such as $\lambda$ and $\beta$ used in MTT + MIM4DD and BPTT+MIM4DD? - Please also see my comments in weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- The derivation seems incorrect - The method introduces strong scalability limitations such as the number of negative pairs and the per-layer matching loss.

Results of this manuscript in Tab1

'Furthermore, our method consistently outperforms all 12 recent DD methods, emphasizing its efficacy and relevance in the current landscape.' Based on the authors' reply, I double check the results in Tab.1. The proposed method does not perform better than TESLA and FRePo-w in some settings. Here are some state-of-the-art methods: HaBa(Neurips2022), IDC(icml2022), DREAM(iccv2023), FTD(cvpr2023). pleased compare with these methods, especially on CIFAR10/100 and TinyImageNet datasets. The sota results in CIFAR10 ipc1,10, and 50 should be around 50.5, 69, 74.5; CIFAR100 ipc1, 10, 50 should be around 29, 45 and 52; TinyImageNet should be around 10, 24, 29. For the hardware issue about IPC50 on cifar100, please refer to the GitHub of DC. The author has shown how to implement it for such large categories, I remember. 'It's essential to highlight that while DC, DM, and DSA were indeed state-of-the-art two years ago, the landscape has evolved. Recent works like MTT (CVPR2023) and BPTT (NeurIPS2022) now represent the new state-of-the-art in this domain.' MTT is proposed in CVPR2022, not 2023.

BPTT + MIM4DD is lower than BPTT?

Hi, thanks for the authors for the response, I have a few other questions 1. BPTT + MIM4DD performs worse than BPTT? The results of BPTT[11] for MNIST IPC 1, 10 and 50 are: 98.7, 99.3 and 99.4. The results reported in this paper for BPTT+MIM4DD is 95.8, 98.9 and 99.2 which are lower than the original methods. Similar downgraded performances are also seen in other datasets such as CIFAR-100, 34.0 and 42.9 for BPTT and 25.0/38.5 for BPTT+MIM4DD. What's the cause of the absolute 9% performance drop? Does it mean that MIM4DD can actually hurt BPTT's performance? There is also a huge 15% performance drop on CIFAR-10 IPC 1 and 10. 2. Can the author upload the results for TinyImageNet, just IPC 1 and 10 are fine if the authors have trouble getting results for CIFAR100 IPC50. It should be quick to run. (BPTT has TinyImageNet IPC 1 and MTT has IPC 1, 10 and 50). It will be great if you can provide the results for IPC 1 for BPTT and MTT and IPC 10 for MTT. 3. What is the scale of $L_{NCE}^k$ loss and $L_{DD}$ loss? Are they on the same magnitude. 4. What is the motivation for dividing $L_{NCE}^k$ loss by $\beta^{K-1-k}$? 5. When you tried to get the results for reviewer: 7GDB, did you also apply the division factor as mentioned in point 3 above? 6. Conflict (typo) in the newly uploaded pdf (figure 1). The figure description says the right one is without MIM4DD and the left one is trained with MIM4DD but the subtitle says the other way around.

Response to Rebuttal

I would like to thank the authors for addressing the comments. I do not have additional concerns. Besides, I must show appreciation to the authors for being so patient and polite when they replied to the apparently stupid question (Question 2). I really don't understand why I even had that confusion from the first place. The rating (7) still reflects my assessment of this submission and I decide to keep it as is. Best of luck.

Please look at the authors' reply

Dear Reviewers, Please do look at the authors' rebuttal if you have not done so. Please let the authors know if they have addressed your concerns. Thanks for your contribution to NeurIPS. AC