Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective

Summary

The paper addresses the dataset condensation task and proposes a new framework termed Squeeze, Recover and Relabel. In this three step approach, the authors first train a model from scratch to accommodate most of the crucial information from the original dataset. In the second stage, target data is synthesized from Gaussian noise. And in the final stage, the generated synthetic data is relabelled using a crop-level scheme to align with the true label of the data. Extensive and controlled experimentation showed significant performance improvement compared to previous state-of-the-art methods.

Strengths

This publication has several strengths including: 1) The writing is clear and easy to understand. 2) Generalizability of the framework to scale of datasets, input resolution, and the size of network architectures. 3) Good experimental methodology with carefully designed ablations that justifies architectural design decisions especially impacts of squeezing budget, diverse data augmentations for original data compression, recovery budget and regularization terms for data recovery and insights on model choice and training for relabeling process. 4) Very exhaustive in-depth empirical comparison with state-of-the-art methods demonstrating strong performance as well as incur reduced compute and memory consumption.

Weaknesses

I am confused by the claims in the section “Cross-Architecture Generalization”. The authors attribute the suboptimal performance of DeiT-Tiny on the condensed datasets due to the model’s inherent need for substantial training data. However, from Table 5 there appears to be a “cross-architecture gap”: ResNet based evaluation models perform poorly on condensed data based on ViT based squeezed model compared to when evaluated on condensed data based on ResNet based squeezed model. Similar observation for DeiT-Tiny evaluated on condensed data based on DeiT-Tiny-BN and ResNet-18 squeezed model. This shows that the condensed does not generalize well to different network architectures.

Questions

The paper in its current form needs clarification on Cross-Architecture Generalization. Please refer to the “Weakness” section for details. The authors acknowledge the performance disparity between condensed dataset in the limitation section. I feel that this might limit the adaptability of the proposed approach. Minor typo: 1) Line 277 refers to Figure 3, it should refer to Table 5. 2) “Deit” -> “DeiT”

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors discuss the limitations in the paper.

Summary

This paper proposes a 3-step dataset condensation approach. Instead of applying bilevel optimization based approach in the previous work, the proposed method break down 3 decoupled steps: squeeze, recover and relabel. The key idea is decoupling the modeling training on real data and the generation of the synthetic data. At the first squeeze step, the model is trained on the original full dataset. At the second recovery step, synthetic data is generated using pretrained model and class prior with additional regularization (TV loss and BN consistency). At the 3rd stage, soft labels are generated using the pretrained model. And finally, a model on synthetic data is trained using the images from step 2 and labels from step 3.

Strengths

Compared with previous method on Tiny-IN & IN-1K, the proposed approach achieves superior performance. The proposed method generates more visually appealing images from the example images in Fig 4. The first work condenses the full IN-1K, based on the claim from the paper. Efficient way of using decoupled steps during the generation.

Weaknesses

1. Lack of strong technical novelty. The proposed method combines a few prior work on image synthesis, such as deepDream & Inverting Image, and applies it directly into a new problem. 2. Lack of comprehensive study with baseline approaches, and demonstration of the improvement of the key novelty.

Questions

I am not familiar with the field, but I am not convinced about the overall problem being solved. If we would like to condense the original dataset for efficient training, we should at least achieve comparable accuracy on retraining. This is not the case from the IN-1K result. If we would like to obtain a way of retraining a new model without requiring original labeled dataset, semi-supervised learning on a unlabeled data would be a more promising direction instead of image synthesis.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please see my concerns in the Questions section

Summary

This paper proposes a new dataset condensation termed Squeeze, Recover, and Relabel that decouples the bilevel optimization of model and synthetic data during training. Extensive experiments show the effectiveness and efficiency of the proposed method in several IPC settings.

Strengths

1, The paper is well-written and easy to understand. 2, The proposed method becomes efficient due to the decoupled stages. 3, The extensive experiments show the effectiveness of the method.

Weaknesses

1, Albeit the computation and memory efficiency in the proposed method, the whole training time might be comparable to other methods like DM and MTT. 2, Unclear illustration about BN layers in VIT. Is it means that the proposed method uses the BN layer to replace the LN? 3, The lack of comparison between the IPC and directly sampling the same scale images.

Questions

1, One naive question: why the condensed images could be used for training? I am not familiar with this topic. 2, The stage-3 Relabel seems like the process of knowledge distillation. Could the author try another setting, like using R50 to teach R18?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a dataset distillation or dataset condensation method that can support ImageNet-scale compression. The main idea is inspired by some data-free knowledge distillation techniques to optimize the cross-entropy error, BN statistic distance, and some other prior terms for the distilled data. The complexity is simple compared with recent mainstream methods of dataset distillation. And the proposed method achieves impressive accuracy for models trained with distilled data on large-scale datasets.

Strengths

1. The proposed method is simple yet effective, which enjoys satisfactory scalability. 2. It is good to know for the community that dataset distillation could achieve promising results on large-scale datasets like ImageNet1k-224 resolution. 3. The writing is coherent and it's easy for readers to follow the proposals.

Weaknesses

1. I am a little bit worried about the technical novelty. Since the main idea is largely inspired by data-free knowledge distillation techniques [a], I list this as a weakness for NeurIPS, the top-tier machine learning conference. 2. Some detailed ablation studies are expected: * I notice that the settings of this paper is different from previous works. For example, previous works typically use Convent for dataset distillation while this work mainly considers ResNet. I am not arguing that the setting must be the same. However, I do think that an ablation study is necessary to show the improvement coming from the architectures. * A sensitivity analysis with respect to $\alpha_{BN}$ is necessary. * I do understand that the proposed method is mainly for large-scale datasets. But it is also interesting to dynamically increase the size of datasets and compare the performance with the existing methods, to help readers better understand when the proposed method yields advantages. [a] Dreaming to distill: Data-free knowledge transfer via deepinversion. Yin et al., CVPR 2020.

Questions

My major questions focus on the detailed ablation studies. Please refer to the 2nd point of the above weaknesses part for details.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors should clarify more limitations of the specific solution they proposed instead of those general issues of dataset distillation, e.g., larger datasets and performance gap with models trained on full data.

Summary

The paper introduces a new dataset condensation framework, that is Squeeze, Recover, and Relabel (SRe2L). SRe2L decouples the optimization of model and synthetic data during training, enabling effective condensation across varying dataset scales, model architectures, and image resolutions. The authors mention in the paper about the advantages such as arbitrary resolution synthesis, low training cost and memory consumption, and scalability to different evaluation network architectures. Extensive experiments on Tiny-ImageNet and full ImageNet-1K datasets demonstrate its improved performance compared to state-of-the-art methods. SRe2L also outperforms the MTT approach in terms of speed and memory consumption during data synthesis. Overall, SRe2L presents a powerful solution for dataset condensation with improved performance and efficiency.

Strengths

1. Clear Paper Organization: The paper exhibits a well-structured organization that aids in comprehending the presented concepts, methodologies, and experimental results. The logical flow of information allows readers to follow the paper's contributions easily. 2. Novel Framework: The authors propose a new dataset condensation framework, Squeeze, Recover, and Relabel (SRe2L), which offers a fresh perspective on addressing the data condensation problem. While some technical details may not be entirely novel, the paper presents an alternative solution to data condensation, introducing new ideas and approaches. 3. Surprising Performance: The scalability of data condensation to large datasets and deep networks poses significant challenges. The paper's performance on ImageNet-level datasets is particularly impressive, demonstrating the effectiveness and robustness of the proposed framework in handling complex datasets and achieving high validation accuracy.

Weaknesses

1. Limited Theoretical Analysis: While the paper presents impressive empirical results and demonstrates the effectiveness of the proposed framework, it lacks a comprehensive theoretical analysis. More theoretical analysis, such as error bounds or performance upper bounds, would provide a deeper understanding of the underlying principles and limitations of the proposed approach. Incorporating theoretical analysis could further strengthen the paper's contributions and provide insights into the algorithm's behavior and performance guarantees. Overall, the paper is well-structured, introduces a novel framework, and achieves remarkable performance on challenging datasets. However, enhancing the theoretical analysis would add a valuable dimension to the paper and provide a more comprehensive evaluation of the proposed method.

Questions

see weakness

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

None

Response to the Rebuttal

I would like to express my appreciation to the authors for their detailed and insightful response to my review. Their effort in providing additional theoretical support has indeed addressed my initial concerns and has significantly enriched my understanding of the paper's contributions. The authors' response has provided a clear and compelling explanation of how their approach fits into the broader landscape of related methods. The clarification regarding the theoretical underpinnings of their method, along with the comparisons to existing techniques, has solidified my confidence in the novelty and importance of their work. I commend the authors for their diligence and thoughtful responses, and I look forward to the continued refinement and impact of their work in the field.

Thanks for the rebuttal!

Thanks the author for the informative rebuttal! It clears most of my concerns. Here are some remaining ones: 1. The authors have provided some results on down-sampled ImageNet datasets to illustrate the performance on small datasets, which conducts the ablation of dataset sizes via changing resolutions. I am curious about the studies with respect to changing the number of images and the number of classes. For example, how about the performance on small-scale datasets like CIFAR? The authors have mentioned in the supplement that the method is for large-scale datasets and the results on small datasets are not competitive. I definitely understand this. Nevertheless, I think it is important to report these results or some other forms of ablation studies on the number of images to help readers understand what sizes of datasets can benefit from this method. 2. I notice that the authors use a trick of "Multi-crop Optimization", which is related to some recent works on synthetic data parameterization [a, b, c], and the performance is indeed sensitive to this operation. The authors are encouraged to have a discussion with these related works. [a] Dataset Condensation via Efficient Synthetic-Data Parameterization (Jang-Hyun Kim et al., ICML 2022) [b] Dataset Distillation via Factorization (Songhua Liu et al., NeurIPS 2022) [c] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks (Zhiwei Deng et al., NeurIPS 2022)