Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation in dataset distillation. Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at https://github.com/shqii1j/seqmatch.
Paper
Similar papers
Peer review
Summary
This paper proposes a novel dataset distillation method called SeqMatch which focuses on extracting high-level features from later training trajectories. The authors highlight a limitation in state-of-the-art data distillation methods, which tend to condense low-level information from easy data while overlooking the high-level information contained in hard data. In response to this issue, the paper introduces a novel optimization technique that generates multiple small sets of synthetic data. Each of these sets distills distinct knowledge from various stages of the training trajectories. By addressing the inherent problem observed in previous dataset distillation methods, the authors conduct experiments to showcase the efficacy of SeqMatch.
Strengths
1. The idea of condensing different sub-datasets for different stages is novel and interesting. 2. The proposed method does not require extra computation cost. 3. The writing is good and easy to follow.
Weaknesses
- The paper lacks evaluation numbers for certain settings, such as SeqMatch-IDC, on datasets like CFAIR100 50IPC, Tiny-ImageNet, and ImageNet subset. (SeqMatch-IDC seems to be the most favorable setting) This omission makes it difficult to determine the performance of SeqMatch in comparison to other methods in these specific scenarios. - SeqMatch underperforms the baseline method FTD on Tiny-ImageNet and ImageNet subset, suggesting that SeqMatch may not scale well to larger datasets. This raises the question of why SeqMatch was chosen over FTD in these cases. - The paper lacks an ablation study, which would provide valuable insights into the impact of the number of subsets, K, on the performance of SeqMatch. Including such an evaluation would enhance our understanding of how SeqMatch operates and how different parameter settings influence its performance. - The caption of Table 1 is somewhat misleading. Although IDC is not categorized as a factorization-based method, it does employ data parameterization (factorization can be treated as a special data parameterization). Therefore, it would be more appropriate to compare IDC with RTP and HaBa rather than other methods distilling information into a single image. - Minor: There are citation inconsistencies in the appendix. For example, FTD is referred to as [12] in the appendix but as [11] in the main text. Additionally, the appendix lacks a reference section, making it difficult to trace the sources of the cited works accurately. - The reproductivity checklist is chosen as Yes, but no code is provided.
Questions
How is the repeated evaluation conducted to calculate the error bar? The motivation of the paper is that SOTA methods fail to distill high-level information from hard data. From my perspective, a more natural and straightforward solution would be to divide the original training dataset into different subsets based on data difficulty and then distill distinct datasets based on these subsets. I am wondering if the authors explored this approach or considered it as an alternative solution in their research.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Yes, the authors have discussed the limitations.
Summary
This work proposes a change to existing dataset distillation methods by sequentially optimizing different subsets at a time. At each iteration, the existing subset is frozen and a new subset of data is *added* to it and optimized. This method allows different subsets of the synthetic data to capture different levels of features required by a network to learn during training time. This method boosts the state of the art for subsets with >1 IPC (since this method does not make sense with 1 IPC).
Strengths
I like this paper a lot. The authors addressed an obvious problem with existing dataset algorithms (all samples capture the same level of features) in an elegant way that is adaptable to all existing and future backbone methods. The authors both empirically and theoretically show that jointly optimizing the entire set couples the gradients in a way that prevents the synthetic set from learning the necessary variety of features. The visuals are all very nice, clearly illustrating the authors' points.
Weaknesses
Algorithm 1 is a bit confusing to read. According to line 4, it seems that a single network initialization is used to optimize the entire subset (but this can't be true since it would catastrophically overfit). It is also unclear what the $n$ parameter is. As the algorithm is currently presented, I cannot see how MTT can be slotted into it. Maybe it would be clearer if an additional inner loop was included along with a generic distillation loss and doing away with the sum over $m$?
Questions
I would like for the authors to make Algorithm 1 more clear as described above. It would also be nice to see visuals when there is just 1 sample per subset (i.e., K=IPC) even with just IPC 2 or 3 since this case would likely have the largest differences between the subsets. Lastly, I recognize his surname is quite long, but Figure 1 should probably cite "Cazenavette et al." rather than "George et al." :)
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
yes
New findings inspired by your comments
Dear reviewer QS3g, We deeply appreciate your constructive suggestions and thorough review of our work. Drawing inspiration from your comments, >start to distill samples that resemble the "harder" or "long-tail" samples from the original dataset We hypothesize that the synthetic data within S3 (representing hard data) of SeqMatch-MTT can serve as a condensed subset of challenging instances, capable of being utilized to 'concentrate on a sparse set of challenging examples,' similar to the approach of FocalLoss[55]. This potential utilization could enhance the performance of a standard training procedure using the original dataset. Consequently, we conducted experiments to empirically test this hypothesis. |ResNet-18|Original Dataset | Original Dataset + S3 of SeqMatch-MTT| |:-:| :-: |:-: | |CIFAR10|95.74|96.38 (**+0.64**)| |CIFAR100|78.05|78.64 (**+0.59**)| The experimental results **align with our expectations**; S3 enhances the standard training of the original dataset by **0.64%** and **0.59%**. This investigation could be a potential application of SeqMatch, to operate in reverse, further amplifying the performance improvement achieved through standard training. In details, the standard training employs basic data augmentation and SGD optimizer(0.05 learning rate, 0.9 momentum, cosine annealing lr, 5e-4 Weight decay) for 200 epochs. The batchsize of original dataset is 128, we randomly sample 13 instances from S3 and integrate with the original mini-batch in each iteration (128 original +13 SeqMatch S3). Best Regards The authors [55] Lin, Tsung-Yi, et al. "Focal loss for dense object detection." Proceedings of the IEEE international conference on computer vision. 2017.
Summary
This paper propose a new method called sequential subset matching (SeqMatch) for dataset distillation. The proposed method is designed to continuously generate synthetic images at different training (distillation) iteration. This strategy is inspired by the general mechanism of optimization, which captures characteristics (low-level feature) of easy instances in an early stage, but takes characteristics (i.e., high-level feature) from increasingly difficult instances. SeqMatch was applied to various dataset distillation methods and showed marginal but better performance than the method in which SeqMatch was applied or other baseline methods in various four datasets.
Strengths
1. The analysis about the general mechanism of optimization is so insightful that it deserves a lot of attention in other studies. 2. The plots of several figures provide good support for the arguments in this paper. e.g., Figures 1 and 2 well illustrate the effect of the general mechanism of optimization on dataset distillation and the coupling issue, respectively.
Weaknesses
First of all, I don't understand the motivation behind designing some SeqMatch. 1. How was the claim for the optimization mechanism that captures low-level features in the early stages and high-level features in the later stages verified? Figure 1 seems to have been used to verify this claim, but can hard instances and easy instances represent high-level and low-level features, respectively? 2. I don't understand how the analysis of the general mechanism of optimization was used to design the SeqMatch method. In particular, what is the motivation for applying the SeqMatch method when training f_theta in the evaluation phase? In addition, in Table 1, most of the performance improvements acquired via SeqMatch are very marginal (~0.2). Performance improvement is not an absolute determining factor in judging the superiority of the proposed method, but it can be used to verify that the method works as claimed in this paper.
Questions
1. In "Gradient Matching Methods" in Section 3, are the gradients from g_1 to g_M the gradients from different layers of a network? Or gradients from another training iteration?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
3 good
Contribution
3 good
Limitations
This paper adequately addressed the limitation and promised to solve it in future work.
A request for reviewing the rebuttal
Dear reviewer Qidh, We appreciate the time you dedicated to reviewing our submission. Recognizing that you may have a busy schedule, particularly when evaluating areas outside your own research field, we kindly request that you consider investing some time to explore more about dataset distillation. This emerging and pivotal machine learning task holds significant importance. We hope you can engage in a comprehensive review of both the rebuttal we have submitted and the insightful comments provided by Reviewer QS3g, an expert in the field of dataset distillation. Your constructive feedback is immensely valuable to us, and we sincerely thank you for your reviewing. Best The authors
Dear Reviewer Qidh, This is another friendly reminder to acknowledge that you have read the rebuttal and the other reviews. Please also share how they change your view on the paper, if at all. Thanks again for your service! Best, AC
Sorry for the late reply. I'm late reading some of the previous work on dataset distillation. Most of my concerns have been resolved, so I will raise my initial rating.
Dear reviewer Qidh, Thank you for your feedback during the review of our paper. We have incorporated your suggestions and rephrased lines 146-148 in Section 3 to enhance clarity. We have also carried out additional experiments, as detailed in A2 for reviewer QS3g and in A3, A6, A7, A9 for reviewer xCSw, in order to provide **a comprehensive evaluation** of our proposed SeqMatch. Thank you once again for your timely response. Best The authors
Summary
This paper investigates an issue with dataset distillation, where synthesized datasets tend to overly condense low-level features but fail to efficiently incorporate high-level ones. The authors argue that this is due to existing methods treating the synthetic dataset as a unified entity and equally optimizing each instance, leading to a coupling issue as the size of the synthetic dataset increases. To address this problem, they propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch). SeqMatch divides the synthetic dataset into multiple subsets and optimizes them in sequence, mimicking the learning process from low-level to high-level features. This approach allows each subset of the synthetic dataset to progressively capture more complex, high-level features, reducing the coupling issue and enhancing overall performance.
Strengths
This paper is well-written, well-motivated, and well-organized. The authors provide comprehensive experiments on various datasets such as CIFAR-10, CIFAR-100, TinyImageNet, and subsets of the ImageNet. They provide insightful analysis of the experimental results, discussing the impact of different factors. The authors provide a detailed algorithm description that translates their theoretical insights into practical application.
Weaknesses
1. It's not clear how generalizable these results are to other tasks and datasets. Their experiments are based on specific DNN architectures and datasets, and the proposed method's effectiveness might vary under different conditions. 2. Since the method divides the synthetic dataset into subsets and optimizes them sequentially, there might be a risk of overfitting, especially when the number of subsets is high or when subsets are small. 3. The method proposed requires a series of sequential optimization processes, which could potentially increase the computational cost and time required for training.
Questions
1. How do you account for the randomness in the initialization of the synthetic dataset, and how does it affect the rate of convergence of each synthetic instance? 2. The authors mention that instances sharing a similar initialization within the same class will converge similarly. Could this lead to any form of bias in your findings? 3. The authors identified that current gradient matching methods prioritize easy instances during early epochs. What is an effective mechanism for identifying or quantifying the 'difficult' instances to create a more balanced emphasis?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
In section 5.4, the authors discussed two limitations of their work. Their openness in acknowledging these limitations adds to the credibility of their research and provides useful guidance for follow-up studies in this field.
A request for reviewing the rebuttal
Dear reviewer u5kd, Thank you for your advice and questions regarding the review of our paper. Serving as a reviewer greatly contributes to the advancement of deep learning research, and it is a crucial duty for researchers. We understand that reviewers have busy schedules, but we sincerely hope that you can spare a moment to review the rebuttal we have submitted to address your concerns. We kindly request you to thoroughly review our rebuttal along with the comments from the reviewer QS3g. Once again, we extend our gratitude for your assistance and hope that we can address all of your questions before the end of the discussion period. Best The authors
Thanks for addressing my concerns and I appreciate the authors' thorough rebuttal. I would like to raise the score to be 5: Borderline accept.
Dear review u5kd, We express our gratitude for your valuable feedback and your dedicated service as a reviewer within the research community. Best The authors
Response to Rebuttal
Thank you for answering my questions, I raise my score to an 8. The visuals for K=IPC are particularly interesting; several classes have very stark differences between images. For example, looking at the frogs, the first image can almost be thought of as an "average" frog. The next biggest performance boost can then be gained by "learning about" bright green frogs. After that, the next biggest performance boost can be gained by "learning about" brown frogs. I wonder if this process were repeated for more IPC if we would start to distill samples that resemble the "harder" or "long-tail" samples from the original dataset, like one of the red, yellow, or blue frogs you can see [here](https://knowyourdata-tfds.withgoogle.com/#dataset=cifar10&filters=default_segment.cifar10.label.value:frog).
Thanks for the comments and raising the score
Dear reviewer QS3g, Thank you for your prompt response! We have taken note of the color variations among synthetic instances from various sequential subsets. Drawing from your insights, we would like to propose a hypothesis: could enhancing the "mutual orthogonality" of the synthetic dataset potentially contribute to improved overall performance? This notion stems from considering each synthetic instance as a fundamental "basis" element. In an effort to substantiate this hypothesis, we are contemplating extending the application of SeqMatch to scenarios involving larger ipc values (>50) and a higher number of subsets (K). Such an expansion could serve as a means to test the hypothesis. Your feedback has prompted us to embark on this exploration. Best regards The authors
A request for reviewing the rebuttal
Dear reviewer xCSw， Thank you for your time and dedication to the review process. We are writing to kindly request your response to the rebuttal we submitted in response to your valuable feedback. We understand that reviewers have busy schedules. However, we are eagerly anticipating your constructive suggestions to ensure the effective resolution of any misunderstandings. With only 4 days remaining in the discussion period, we are concerned that there may not be sufficient time to further clarify any misconceptions in our submission. If you could spare a moment to review our rebuttal and share your thoughts, it would immensely help us enhance the quality of the submission. We kindly ask for your comments on any remaining concerns you may have about our submission. Your assistance in this matter would be greatly appreciated. We truly value your input and eagerly look forward to your response at your earliest convenience. Best The authors
Thanks for the response
Thanks for the response. The reason that I replied a bit late is that I am still waiting for the evaluation results promised in A1, but it seems that it is not ready yet. I appreciate the effort put into addressing my previous concerns. However, after considering the response and taking into account other reviews, I still have the following concerns: 1. I am waiting for the evaluation results that the authors promised in A1. 2. As mentioned by other reviewers, the improvement is not so significant -- sometimes less than (1%), which hinders the contribution of this paper. 3. I find a misalignment between the stated motivation and the methodology. SeqMatch does not directly address the challenges discussed in the motivation: distilling high-level information from hard data. Minor in response: IDC is a data parameterization method but not a factorization-based method. Based on the above concerns, I am inclined to keep my rating unchanged.
2nd Response (1/2)
We appreciate your continued feedback. We anticipate engaging in further discussions and would like to provide the following clarification at this point. **Q7:** the evaluation results promised in A1, but it seems that it is not ready yet. **A7:** The complete results are not ready as the training speed of IDC[21] in cifar100 with ipc=50 is slower than we expected. Because we have to reduce the synthetic batchsize from 64 to 40 due to the huge memory cost caused by the multi-formation technique of IDC[21]. We present the current results at $4713^{th}$ ($6000$ in total) iteration as below, ||S1 |S2 |S3 |Entire| |:-:| :-: |:-: |:-:|:-: | |Haba|-|-|-|47.0| |IDC|-|-|-|51.4| |SeqMatch-IDC|46.1|48.9|51.0 (713/2000)|51.0 (4713/6000)| Our proposed **SeqMatch-IDC** is currently approaching the baseline IDC with 1287 itertions to go. --- We also embed our proposed SeqMatch with the baseline FTD[11]. We evaluate **SeqMatch-FTD** on the two subsets of ImageNet, ImageNette and ImageFruit, we report the results as below, | |ImageNette|ImageFruit| |:-: | :-: |:-:| |MTT[5]|63.0|40.3| |Haba[32]|64.7|42.5| |FTD[11]|67.7|44.9| |SeqMatch-FTD|**70.6** (+2.9)|**46.5** (+1.6)| SeqMatch-FTD enhances FTD with an average increase of **2.25%** in performance. Notably, SeqMatch-FTD showcases a notable enhancement of **2.90%** on the ImageNette subset, underscoring SeqMatch's superior performance on larger-scale datasets. The ongoing experiments on the ImageWoof and ImageMeow subsets are currently in progress. --- **Q8:** the improvement is not so significant **A8:** MNIST and SVHN are both digit recognition datasets where existing dataset distillation methods could achieve a marginal performance drop after distillation. However, SeqMatch-IDC still achieves a 92.1% accuracy on SVHN with ipc=50, which outperforms other baselines with an accuracy increase of more than **1.90%**. On the CIFAR datasets, SeqMatch has acquired an **averaged** performance improvement of **2.25%** over the baseline MTT[5] and **0.63%** over IDC[21]. The corresponding **averaged** performance improvement on ImageNet Subsets are **3.88%** over MTT[5] and **2.25%** over IDC[21]. Of greater significance, SeqMatch functions as a training strategy capable of seamless integration into established dataset distillation methods. Our experiments involving the incorporation of SeqMatch into three advanced baseline approaches, namely MTT[5], IDC[21], and FTD[11], serve as validation for the efficacy of our proposed SeqMatch. --- **Q9:** SeqMatch does not directly address the challenges discussed in the motivation: distilling high-level information from hard data. **A9:** We begin by demonstrating the **existence of the challenge** through experiments depicted in Figure 1. In these experiments, the synthetic dataset, designed to encapsulate information from the original dataset, exhibits **a failure to reduce the loss associated with hard data** while **overly reducing the loss of easy data**. These outcomes highlight the challenge that current methods encounter in distilling high-level information from hard data. To further validate the impact of SeqMatch, we observe that it effectively **encourages** baseline methods to prioritize the extraction of high-level information from hard data. This is evident as the **loss of hard data** experiences **a significant reduction** in comparison to the vanilla baseline, as depicted in Figure 1. And the **excessive focus on easy data is mitigated**. Furthermore, the heatmap presented in Figure 4 of the Appendix corroborates the same contribution highlighted in Figure 1. Hence, we validate that SeqMatch improves the performance of baselines by promoting the distillation of high-level information from challenging data instances. **We believe our divergence lies in the way to regularize the distillation from hard data.** The most strightforward approach to do so is **only** distilling information from hard data subset, which is as you suggested in $1^{st}$ review. However, our response in A6 confirms the **inadequacy of this approach** due to its poor performance. This outcome is logical since the **vanilla training doesn't differentiate** this hard data subset either. The instance sampling remains consistent throughout the entire standard training process. Simply learning or distilling solely from hard data would lead to a severe issue of **catastrophic overfitting** on hard data. In fact, the **discrepancy across epochs holds a much greater significance** in terms of the density of high-level information compared to **differences at the instance level**. As a result, we opt for regularization in an epoch-wise manner rather than instance-wise. **Each synthetic subset follows distinct learning trajectories across various epochs** (e.g., S1 learns from epoch 0-10, S2 learns from epoch 10-20, and so on). This is also the rationale behind the emphasis on the term "sequential" in naming of our method.
2nd response (2/2)
**minor** IDC **Ans** We refer to the survey paper by Lei et al.[28] (Sec. 5, Para. 2, and Table 1), which categorizes IDC[21] as a factorization-based method. The key point of contention lies in whether the multi-formation function should be classified as a decoder. The survey paper by Lei et al.[28] confirms that the multi-formation function indeed functions as a decoder. However, we concur with your viewpoint that IDC[21] should not be classified as a factorization-based method due to the utilization of the training-free bilinear upsampling algorithm for upsampling the partitioned data in the multi-formation function. Consequently, we will incorporate a specific footnote for IDC[21] in Table 1. [28] Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.05603, 2023. 3, 6, 7, 8
Updating of the experiment results
Dear reviewer xCSw, We would like to update our results on the ImageNet subsets of SeqMatch-FTD as below, | |ImageNette|ImageFruit| ImageWoof |ImageMeow| |:-: | :-: |:-:| :-: |:-:| |MTT[5]|63.0|40.3|35.8|40.4| |Haba[32]|64.7|42.5|38.6|42.9| |FTD[11]|67.7|44.9|38.8|43.3| |SeqMatch-FTD|**70.6** (+2.9)|**46.5** (+1.6)|**41.1**(+2.3)|**45.4**(+2.1) Overall, SeqMatch-FTD achieves an average performance enhancement of **2.225%** over FTD[11] across the four ImageNet subsets. The results confirm the **consistent performance** of SeqMatch in **larger-scale datasets**. We still do not finish the experiments of SeqMatch-IDC on CIFAR100 with ipc=50 due to the complex three-level nested loop structure of IDC (4.2 mins/iteration). We update the results below, ||S1 |S2 |S3 |Entire| |:-:| :-: |:-: |:-:|:-: | |Haba[32]|-|-|-|47.0| |IDC[21]|-|-|-|51.4| |SeqMatch-IDC|46.1|48.9|51.7 (1256/2000)|51.7 (5256/6000)| Despite having **744 iterations remaining for training**, SeqMatch-IDC has already surpassed IDC[21] by **0.3%**. We are confident that SeqMatch-IDC's performance will **further improve** as the training progresses. We are eager for the opportunity to address any remaining concerns you might have before the discussion period concludes. It's worth mentioning that **both reviewer QS3g and reviewer u5kd** have found our responses **satisfactory**, leading them to **raise the score** of our paper. Upon acceptance of our paper, we will release the codes and the distilled datasets. We sincerely appreciate your and other reviewers' dedication and time invested in reviewing our work. Best The authors
New findings to answer Q9 and updating of experimental results
Dear reviewer xCSw, We would like to provide a final update on SeqMatch-IDC's performance on CIFAR100 with ipc=50, as shown below: ||S1 |S2 |S3 |Entire| |:-:| :-: |:-: |:-:|:-: | |Haba[32]|-|-|-|47.0| |IDC[21]|-|-|-|51.4| |SeqMatch-IDC|46.1|48.9|51.9 (1541/2000)|51.9 (5541/6000)| SeqMatch-IDC has already surpassed IDC[21] by **0.5%**. Despite the experiments in Figure 1 of our paper and the clarification of A9 in second respose, **we also conduct another experiments to address your concern on Q9.** >**Q9:** SeqMatch does not directly address the challenges discussed in the motivation: distilling high-level information from hard data. Our experiments verify that the synthetic data within **S3 of SeqMatch-MTT** could be integrated with the original dataset to **enhance the performance of a standard ResNet18 training** on CIFAR10/100 datasets, which is shown below, |ResNet-18|Original Dataset | Original Dataset + S3 of SeqMatch-MTT| |:-:| :-: |:-: | |CIFAR10|95.74|96.38 (**+0.64**)| |CIFAR100|78.05|78.64 (**+0.59**)| Drawing inspiration from FocalLoss [55], which assigns lower weight to well-classified examples (easy data) and focuses on a sparse set of challenging examples (hard data), we **incorporate the S3 of SeqMatch-MTT** into a standard ResNet18 training to **strengthen the impact of hard data**. If S3 effectively distills high-level information from hard data, this integrated training should improve performance similarly to how FocalLoss [55] operates. The experimental results **align with our expectations**; S3 enhances the standard training of the original dataset by **0.64%** and **0.59%**. Furthermore, the experimental outcomes provide **additional evidence that SeqMatch indeed promotes the extraction of high-level information from hard data**. Notably, SeqMatch has the capability to operate in reverse, further amplifying the performance improvement achieved through standard training. In details, the standard training employs basic data augmentation and SGD optimizer(0.05 learning rate, 0.9 momentum, cosine annealing lr, 5e-4 Weight decay) for 200 epochs. The batchsize of original dataset is 128, we randomly sample 13 instances from S3 and integrate with the original mini-batch in each iteration (128 original +13 SeqMatch S3). The authors [55] Lin, Tsung-Yi, et al. "Focal loss for dense object detection." Proceedings of the IEEE international conference on computer vision. 2017.
Thanks for the comments and raising the score
Dear reviewer u5kd, We would like to thank you for the valuable advice and for raising the score of our paper. We will incorporate your suggestions, especially regarding the effective mechanism that **quantifies and reinforces** the 'difficult' data in the dataset distillation process. Best The author
Decision
Accept (poster)