Frequency Domain-based Dataset Distillation

Summary

This paper proposes a frequency-domain dataset distillation approach that can help to select the most important information in images to help distillation. To do this, the approach starts with a DCT to the input $x$ and then performs loss matching using existing dataset distillation approaches. Experiments demonstrate its advantages over state-of-the-art approaches.

Strengths

1. The idea of using frequency domain information is novel and helpful. 2. The experiments are good and solid. 3. The visualization helps to understand the algorithm and results better.

Weaknesses

1. Experiments are solid, but not enough. I think that dataset distillation is useful for large volume datasets. But experiments do not contain any large datasets. So, the experiments are not useful to help interprete the usefulness of the algorithm. 2. There are no transfer experiments. Since dataset distillation is used to store a proxy of large datasets, and large datasets are often act as the source domain for transfer learning. Therefore, authors should conduct experiments on transfer learning datasets to show the advantage of the proposed approach. Only doing experiments on CIFAR-level datasets are not enough since to be honest, CIFAR is small and does not need to distill. 3. Any complexity analysis? How efficient is the algorithm?

Questions

See weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weakness

Summary

The paper proposes a novel way (code named FreD) to perform parameterization on dataset distillation tasks. Different from other methods which all operate on the spatial domain, this paper proposes to convert the spatial domain into frequency domain which are more concentrated and then perform distillation on these concentrated frequency components. Empirical evaluations show that FreD is able to achieve SOTA compared to previous methods. ============ I have read the author's responses. Part of my concerns are addressed such as the questions regarding IPC while some others remain debatable. Overall, I would still keep my original score. ============

Strengths

## originality - the paper proposes an interesting new way to perform dataset distillation which has not be explored previously to the best of my knowledge ## quality - the paper shows the effectiveness of the proposed methods through an adequate amount of experiments - the method is also theoretically proved through proposition 1 and corollary 1. ## clarity - the paper is well written and easy to follow - the paper categorizes baseline methods clearly which makes the evaluation results easy to follow - the increment of decoded instances are also clearly indicated in table 1 - distilled results are visualized and showed in the paper ## significance To the best of my knowledge, previous parameterization methods mostly perform on the spatial domain. This paper is the first to propose converting spatial domain to frequency domain which concentrates the information better. It could potentially open up a new directions for dataset distillation/condensation methods.

Weaknesses

- The IPCs used in table 1 are 1/10/50, but 2/11/51 in table 2. The motivation for doing so is not clear. - In Figure 2, does the image used come from the evaluation datasets? If not, it's confusing to use an image that is not related to the evaluation datasets in this paper. - The evaluation datasets are all small datasets such as 32 * 32 , in the appendix, only IPC 1 for TinyImageNet is reported. What's the reason for the missing numbers on IPC 10/50? Is it scalability issues or performances issues or something else? - In [1], IPC 1 and 10 are used to evaluate the performance on ImageNet subsets, but IPC 2 is used in Fred, what's the specific reason to choose a different IPC than previous methods? Can we see some results on IPC 1 and 10 as well because it will be good to see how FreD performs on higher resolutions. [1] Dataset Distillation by Matching Training Trajectories

Questions

See my comments on weakness. I am willing to raise the score if the above questions are answered so that I can get a better idea of how FreD works beyond what's currently shown in the main paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- the evaluation in the main paper is only done on datasets with low resolution, thus the scalability is hard to verify - same with a lot of previous parameterization methods, FreD also seem to work best with smaller IPCs such as IPC 1(where generating more images can boost the performance a lot). This can be seen on CIFAR-100 IPC 50 that even with 2.56 times more images, the performance is close to methods without parameterization.

Summary

This work introduces Frequency-based Dataset Distillation as a new means of parameterizing the distilled dataset that requires a much smaller memory footprint. By learning the distilled dataset in the frequency space (defined by the discrete cosine transform by default), a binary mask based on the class-wise EVR can be used to select only certain frequency components with minimal performance tradeoff. The memory saved by masking out these components can be allocated towards distilling more total samples.

Strengths

This work is well written and includes an exhaustive suite of experiments showing the effectiveness of the method. Nearly all questions I had while reading were addressed later on in the paper.

Weaknesses

It is not clear which dataset distillation loss is used in Figure 2. Likewise, it is unclear which distillation loss is used by FreD in Table 1 (although I think it is TM based on info in the appendix). I would recommend removing IPC from the tables all-together and including the number of parameters instead. For example, instead of IPC = 10 for CIFAR-10, you could have #Params = 30,720 (10x3x32x32). IPC can be misleading when comparing to re-parameterization methods since the IPC simply isn't true anymore. Would it be possible to see any visualizations for 3D MNIST? I searched through the appendix but only found visualizations for the 2D datasets. I've been very curious to see dataset distillation done on 3D data.

Questions

I really like the paper overall and would be happy to raise my score if the authors address the few weaknesses described above.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

This paper introduces a new parameterization method for dataset distillation, called FreD, that utilizes the frequency domain to distill a compact, synthetic dataset from a large-sized original one. This approach differs from conventional methods that focus on the spatial domain. It employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD is able to select a subset of frequency dimensions for optimization. The paper provides both theoretical and empirical evidence of FreD's efficiency compared to conventional parameterization methods. In addition, it highlights the compatibility of FreD with existing distillation methods and its consistent improvement in performance over different benchmark datasets. This work addresses the challenges posed by big data and it improves upon the current dataset distillation methods by offering an optimized and memory-efficient solution.

Strengths

1. The authors have done a great job of describing their proposed method, FreD. They offer explicit step-by-step explanations of how each element of the method operates, from the synthetic frequency memory to the binary mask memory and the inverse frequency transform. It helps the reader can follow along with their reasoning. 2. This work introduces an innovative parameterization based dataset distillation method, and is the first work of studying from a frequency perspective. It lays a foundation for further research and discussion in this frequency-oriented parameterization methods.

Weaknesses

1. While this work demonstrates the performance increased relative to several previous methods, it did not compare with another parameterization method [1], where the performance seems to be better than this work. It would be good to include this comparison and discuss the potential limitations of the frequency-based parameterization approach. 2. it's unclear how the choice of transform (Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), or Discrete Wavelet Transform (DWT)) might influence the performance or effectiveness of their approach. This aspect is not thoroughly investigated or discussed. 3. The Spectral Bias could impact the network's ability to generalize to new data, there is little discussion on this which could hurt the effectiveness of the frequency domain-based approach presented in the paper. [1] Deng, Zhiwei, and Olga Russakovsky. "Remember the past: Distilling datasets into addressable memories for neural networks." Advances in Neural Information Processing Systems 35 (2022): 34391-34404.

Questions

1. In section 3.3, can the assumption that the function F is linearly bijective limit the applicability of the method? Is there any situation where this assumption might not hold and how would that affect the results? 2. How does the approach handle high-dimensional datasets? Is there a limit on the dimensionality that the method can effectively handle?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors did not include the limitation section, it will be good to have a discussion on cross architecture generalization ability of the proposed method.

Thanks for reviewer tks8

Thanks for your work on the reviews and the feedback. Please leave a comment if you have further questions. We would be happy to provide additional clarification. Again, Thank you so much for your time.

Thank you for the reviews. Any comments for further clarification?

Dear Reviewers, We truly appreciate your constructive comments on our work and thank you so much for your time. Based on the valuable reviews, we have tried to clarify how our proposed idea would work, its applicability, and its efficacy. Also, we have attached the rebuttal material which contains additional experimental results to support our explanations. We are willing to provide further explanation on any unclear point. Please let us know if there's anything we should have expressed more clearly. Best regards.

Thanks for your response

I would like to thank the authors for your response and most of my concerns are resolved. I will not change my rating to remain a positive side of this paper.

Please look at the authors' reply

Dear Reviewers, Please do look at the authors' rebuttal if you have not done so. Please let the authors know if they have addressed your concerns. Thanks for your contribution to NeurIPS. AC