Data augmentation has been proven effective for training high-accuracy convolutional neural network classifiers by preventing overfitting. However, building deep neural networks in real-world scenarios requires not only high accuracy on clean data but also robustness when data distributions shift. While prior methods have proposed that there is a trade-off between accuracy and robustness, we propose IPMix, a simple data augmentation approach to improve robustness without hurting clean accuracy. IPMix integrates three levels of data augmentation (image-level, patch-level, and pixel-level) into a coherent and label-preserving technique to increase the diversity of training data with limited computational overhead. To further improve the robustness, IPMix introduces structural complexity at different levels to generate more diverse images and adopts the random mixing method for multi-scale information fusion. Experiments demonstrate that IPMix outperforms state-of-the-art corruption robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also significantly improves the other safety measures, including robustness to adversarial perturbations, calibration, prediction consistency, and anomaly detection, achieving state-of-the-art or comparable results on several benchmarks, including ImageNet-R, ImageNet-A, and ImageNet-O.
Paper
Similar papers
Peer review
Summary
This paper proposes IPMix, a data augmentation strategy to improve model robustness (and other safety metrics like calibration). The idea builds upon PixMix, a recent data augmentation strategy generating diversity via synthetic pictures. IPMix performs this at multiple scales (pixel, patch, image levels) and mixes the resulting transformed images. The evaluations are performed on CIFAR-10, CIFAR-100, ImageNet and their shifted versions against several data augmentation baselines.
Strengths
The paper is very well written, it is quite easy to understand. The method is clear and reasonable, given that distribution shifts may happen at different levels it makes sense to address them at different levels. The comparisons and ablation studies are well thought. Additional experimental results in supplementary are also helpful.
Weaknesses
My main concerns are about ImageNet results which suggest minor improvements over PixMix, e.g. ImageNet-C results are 78.1 vs 77.8. The results are close for other benchmarks too like ImageNet-R and ImageNet-P (very close to AugMix). Same goes for anomaly detection results. This makes me wonder: Does using synthetic data and mixing have inherent limitations in terms of generating diversity? For example, what happens if you increase your diversity using recent diffusion models to generate data, e.g. as in [A]? My other questions and comments: - I would be interested in seeing comparisons to DeepAugment as well, as it has been shown to improve robustness notably when combined with AugMix. - How does IPMix perform against natural distribution shifts like ImageNet-V2 and ObjectNet? What about more recent ImageNet benchmarks like ImageNet-3DCC and ImageNet-E? I believe having these evaluations could also provide some insights into the limitations I mentioned above. - When I was reading L201 I thought about the work of Mintun et al. ([15] in supmat) and I was happy to see that you added evaluations on CIFAR-C-bar in the supplementary, which again seems to suggest minor improvements over PixMix. What would be the results for ImageNet-C-bar? - It is nice that you added training overhead in supmat too, but I think it should be better linked in the main paper as I was going to ask about it and it wasn’t pointed out in the main paper. - I understand this is a CNN based study but given the landscape and domination of ViTs I believe evaluations on ViT backbones are also necessary to truly understand the performance of IPMix. [A] Sariyildiz et al. Fake It Till You Make It: Learning Transferable Representations From Synthetic ImageNet Clones, CVPR 2023.
Questions
Please see weaknesses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Please see weaknesses. After rebuttal, I am increasing my rating to borderline accept.
Summary
This work proposes a data augmentation method, named IPMix, which is designed to improve robustness without hurting clean accuracy. Specifically, IPMix utilizes different levels of data augmentation (image-level, patch-level, and pixel-level) to increase the diversity of training data and introduces structural complexity to generate more diverse images. The authors have evaluated the robustness of IPMix to adversarial perturbations, calibration, prediction consistency, and anomaly detection.
Strengths
1. The evaluation of robustness and ablation study is extensive. The authors have comprehensively validated the robustness of IPMix, including adversarial perturbations, calibration, prediction consistency, and anomaly detection. Moreover, the ablation of design choices is in detail. 2. The results on small-scale datasets are promising. Overall, IPMix achieves promising accuracy on CIFAR datasets, as well as safety metrics.
Weaknesses
1. Comparisons with RandAugment [1]. The whole design of IPMix seems to be a complicated combination of multiple existing data augmentation methods and the image-level transformations are important components of RandAugment. According to Tab. 6, these image-level transformations are the most effective modules in the whole design and it is suggested to compare IPMix with RandAugment, which is one of the most popular choices in current training frameworks. 2. Generalization ability to competing methods. Current training frameworks of object recognition [2][3] often utilize multiple data augmentation methods during training, the reviewer wonders whether IPMix is compatible with these methods and could further improve the performance. 3. Results on ImageNet. The accuracy of IPMix on imageNet is not very convincing and the whole results on ImageNet seem strange, as Mixup, CutMix both lead to inferior results compared to the baseline. This phenomenon is contradictory to the results of their papers and the explanation of this part is highly appreciated. [1] Cubuk E D, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020: 702-703. [2] Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11976-11986. [3] Li K, Wang Y, Gao P, et al. Uniformer: Unified transformer for efficient spatiotemporal representation learning[J]. arXiv preprint arXiv:2201.04676, 2022.
Questions
Apart from the points in Weaknesses, the reviewer also has some other questions: 1. Validation across different architectures. The authors have only validated IPMix on ResNet which not be enough to prove the generalization ability of IPMix. 2. Revision on Fig. 4. It is advised to add the names of the three methods from Tab. 2 to Fig. 4 which will be easier for reading.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
N/A
Summary
The authors propose IPMix -- a new data augmentation technique applied during training that improves model robustness across a variety of test-time tasks (e.g., distributional shift, adversarial data augmentation) without compromising accuracy on in distribution tasks (when compared to a model trained in a traditional manner). The acronym IPMix indicates that the technique utilizes augmentations at the Image (I) and Pixel/Patch (P) level and mixes these together with each other and the original image. IPMix augments samples via a convex combination of an augmented and original image, where the novelty in the augmentation operations applied to the image (compared to existing augmentation techniques e.g. Augmix) come from operations between an image and unlabeled synthetic images (e.g., fractals). IPMix research is motivated to produce safe ML systems (e.g., improve model robustness and calibration). **Main Contributions:** * IPMix: A label preserving data augmentation technique that achieves most balanced performance across a variety of tasks desired for safe ML systems (accuracy, robustness, calibration, and anomaly detection). * A thorough empirical study of IPMix using multiple datasets, architectures, and tasks compared to large set of existing data augmentation schemes. **Post-rebuttal update** The authors addressed all of my weaknesses and questions and I have raised my score from a weak accept to accept.
Strengths
* **Experiments**: Experimental results with IPMix compared to existing data augmentation methods (in terms of model accuracy, robustness, and calibration) are impressive and are largely consistent across many datasets and architectures. * **Data augmentation analysis**: Some analysis of models trained with varying data augmentation schemes highlights strengths of IPMix (Figure 8) in comparison to existing data augmentation schemes.
Weaknesses
- **Motivation**: The use of unlabeled synthetic images in the Pixel/Patch mixing did not seem well motivated to me. I agree that IPMix seems to work well based on the experimental results and understand that the choice of unlabeled synthetic images avoids mislabeled augmented training images. However, I feel that some motivation (practical or theoretical) is needed to help clarify and better motivate the design of IPMix and why it would improve performance beyond existing methods (e.g., Augmix) that do not produce augmented images with incorrect labels. - **Paper organization**: A lot of the paper is relegated to the appendix (Appendix F (combination experiments), Appendix E (Fractal generation), Appendix B.1 (IPMix insensitive to mixing set change), Appendix C (IPMix algorithm). I understand space constraints and submission limitations, but I would have liked to see more of the details of IPMix and experiments informing its design in the main body. Perhaps the related work section could have been in the appendix to make space for this.
Questions
1. **Motivation**: What is the motivation for unlabeled synthetic images (e.g., fractals) used in IPMix set? I suspect this is very clear to the authors, but it is not well motivated or explained in the main body. 2. **Experiments**: It is unclear from the paper and appendix if multiple replicates are trained for each architecture/dataset/augmentation configuration (Tables 3, 4, and Figure 6). I am under the impression that a single model was trained for each configuration. If that is the case, it is unclear how much deviation (in accuracy, robustness, calibration) the models trained with varying data augmentation schemes have. I would typically expect some result on this for smaller datasets (e.g., CIFAR-10). Is this something you considered that I missed? 3. **Overhead/cost**: Is there an observable overhead/cost of using IPMix during training (when compared to vanilla training or other data augmentation techniques)? In my experience, Augmix increases training time due to augmentation scheme so I am curious if IPMix is more/less costly than Augmix (as it produces better models than Augmix). 4. **Robustness**: How do models trained using IPMix compare to robustbench leaderboard Common corruptions (i.e., CIFAR-10-C, ImageNet-C)? I suspect they are comparable, but it would potentially strengthen the work by comparing to SOTA models from robustbench. The architectures used in the paper are commonly used on robustbench so it would likely be an easy/straightforward comparison. *In this direction: Did you consider utilizing the Jensen-Shannon divergence (JSD) loss utilized in the original Augmix paper in experiments with IPMix? I suspect this may offer additional improvements to models trained with IPMix so it may be worthwhile to consider if comparing to robustbench models.* 5. **Figure 6**: Is this calibration error for a single architecture trained using each data augmentation technique evaluated on CIFAR-100 (e.g. ResNet-18)? The caption does not specify. Minor suggestions: * Table 1 is provided but never referenced or commented on in the text of the paper. I would typically expect some minor commentary on tables/figures. * Table 1: “Accuracy Error” seems like odd/confusing phrasing to me. Wouldn’t “Classification Error” be clearer?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
This paper proposes a new data augmentation method, IPMix, for vision models that integrates three levels of augmentation techniques (image level, patch level, and pixel level). The approach is label preserving and enforces extra structural complexity to enhance the robustness and other safety measures. Experimental results on multiple open datasets and backbone models are provided to support the research claims in the paper.
Strengths
1. In addition to clean accuracy, the authors seek to optimize both the model robustness and various safety measures in the paper. 2. The provided evaluations are extensive and improvements are observed in multiple performance perspectives compared to the baselines. 3. The paper is generally easy to understand and follow.
Weaknesses
1. Throughout the whole paper, it is kind of hard to understand what is the main technical contribution made in the paper that is original and novel. Although multiple components have been introduced, they seem to be trivial applications and combinations of previously published approaches, since the design of each component of inspired by previous works. Besides, those components are not connected and introduced through a coherent idea but are more like random pieces that are very loosely connected. What main technical challenges have been solved in this paper? If we consider the combination pipeline of different augmentations as the main contribution, it is still trivial because the authors just experimented with different intuitive combination methods in Table 2 and selected the overall best one. 2. From the paper, I don't understand exactly why label-variant augmentations lead to manifold intrusion and lower robustness against unseen data. If the label-preserving property is important, can we enhance the CutMix and MixUp by mixing original images with fractal images and achieving a similar performance as IPMIx? 3. In the evaluation section, the authors only focus on showing the results but do not provide any analysis of the results, which results in the fact that the only piece of information expressed there is IPMix is better than the baselines in various settings, but not about why IPMix works better. 4. The ablation study is not designed in the correct way. Its current form only conveys the message that all three levels of augmentations are needed while removing any one of them will lead to degradations in the performance. In addition to that, the authors should use experiments to show the performance degradation without proper combinations, fractals (label variance), and the pieces in Section 4.2. 5. The authors mentioned the computation efficiency improvement caused by the approach, but no evaluations are performed to test the computation efficiency between IPMix and conventional image-level augmentations. 6. Here are some minor comments: (1) Since the paper depends on image-specific intuitions, it would be good to be precise on the scope in the paper title. (2) In line 183, the \lambda is directly used without introducing its purpose and functionality. (3) The specific design scope and definition of IPMix is unclear. The title of Section 4.1 is about IPMix, while Section 4.2 further introduces more design components. Do they belong to part of IPMix? If so, the organization of the paper should be fixed. (4) The ticks in Figure 6 are too small and hard to recognize.
Questions
I think the main question that should be answered is the main technical contribution made in this paper compared to the existing literature, and what main challenges have been solved.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Yes, the limitations are discussed in the Appendix of the paper.
Summary
This paper presents IPMix, a new data augmentation strategy that enhances robustness without compromising clean accuracy. The proposed approach diversifies the training data with limited computational overhead. IPMix introduces structural complexity at different levels to generate a wider variety of images and employs a random mixing method for multi-scale information fusion to bolster robustness.
Strengths
- The paper addresses augmentation from both a clean accuracy and safety perspective, which is commendable. - The paper is generally well-written and easy to follow. - The figures provided in the paper are informative and well-polished and enhance the understanding of the proposed strategy. - The paper includes many implementation details and ablation studies in Appendix.
Weaknesses
- The paper could benefit from including error bars to account for variance across different seeds, particularly when the performance of each method is comparable. - The paper could provide a more in-depth analysis of why the proposed method improves each safety metric. While the explanations regarding diversity and regularization effects are reasonable, it would be beneficial to understand which part of the pipeline is most crucial for a specific safety measure (e.g., robustness, calibration). The significance of increasing structural complexity using "fractals" could be discussed more. - The title emphasizes the "label-preserving" property, yet it is unclear how vital this is. The authors could discuss the potential outcomes of mixing two images with different labels and linearly interpolating the label. - The experimental focus is primarily on ResNet variants. Including results on the performance of other architectures (e.g., Transformer) could strengthen the paper.
Questions
- There seems to be no discussion on Table 1, which feels disconnected from the rest of the paper. Could the authors elaborate on this? - In Section 4.1 "The IPMix framework", the terms "Chain-Mixed", "linear mix", and "mixed input" are not clearly defined. Could the authors clarify these and possibly include the corresponding framework name in Figure 4 alongside the number? - In Section 3, integration experiments are described. It is suggested that combining previous approaches may result in diminished performance, as the optimal parameter for using each data augmentation independently may not be optimal when used in combination. Could decreasing the augmentation strength of each method yield better performance? - Table 6 suggests that image-level data augmentation is the most crucial part of CIFAR100. Is this observation consistent with ImageNet as well? - How can the advantages of the three methods be leveraged while avoiding their drawbacks? Could the authors provide more specific information about the drawbacks of each method?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors discuss the limitation and societal impact in Appendix.
Please respond to author rebuttals
Dear Reviewers, Please respond to authors after carefully reading the author rebuttals and other reviews. If your assessment of the paper changes, please update your score with a short justification for the new rating. The paper received diverging initial reviews. Please consider discussing with the authors or other reviewers to see whether we can reach a consensus. Thank you, AC
Appreciation and understanding during the rebuttal phase
Dear Area Chair TsJd, Thank you for your attention and promptness. We understand the workload of reviewers and appreciate their time. We sincerely hope the process continues smoothly. Best, Authors
Thanks for the thorough respones.
Thanks for the thorough responses. I have read all the replies, and unfortunately, I can not find enough reasons to accept the paper, and can only raise my rating to 4.
The thanks and further clarification on reviewer Bz96 concerns
Thank you for reviewing our paper and taking the time to read our responses. We respect your feedback. We noticed that the main questions: "The main question that should be answered is the main technical contribution made in this paper compared to the existing literature" might not have been addressed in the depth it deserved in our initial rebuttal due to space constraints. To clarify further on this point, we want to compare IPMix with AugMix and other P-level (pixel-level and patch-level) methods. 1. Compared with AugMix. * **Motivation**: AugMix mixes an image with its versions that have been processed with image-wise (e.g., rotation) methods to improve robustness. However, AugMix has lower accuracy compared to other patch-wise data augmentation methods, as shown in Table 3. Our motivation is to combine the strengths of different levels into a unified and coherent whole, to enhance the **overall** performance. * **Objective**: The purpose of AugMix is to tackle data shifts caused by corruption, but the actual situations encountered in real-world scenarios are much more complex than those it takes into account, as mentioned in the paper [1]. Our objective is to improve performance across multiple safety metrics, including corruption robustness, consistency, anomaly detection, adversarial robustness, and calibration confidences simultaneously, which is more meaningful from a **practical perspective.** * **Method & Framework**: Except for the image-level augmentation, IPMix mixes clean images with synthetic pictures at different scales to improve **structural complexity** and uses random pixels mixing and random elements mixing for better data integration. While AugMix applies augmentation transformations only at the image level, its performance in classifying partially occluded objects is sub-optimal. Furthermore, our goal is to identify the **optimal framework** for generating highly diverse augmented images. The experimental results in Table 2 imply the potential existence of a **general framework** for data augmentation, which we consider a promising avenue for further research in this field. * **Performance**: AugMix is considered a method with state-of-the-art performance in improving model robustness. In comparison with AugMix, our method shows **significant** improvements in CIFAR-10-C, CIFAR-100-C, ImageNet-C, and the other datasets, as shown in Tables 3, 4, and 5 in the main text, which provides compelling evidence of the superiority of IPMix over AugMix. 2. Compared with other P-level methods. * **Method**: We are the first to propose a data augmentation method that mixes fractals with the original image at different level. Compared with existing P-levels works (CutMix, TokenMix, AutoMix, AdaMixUp,etc), our method is **label-preserving**, which is plug-and-play and significantly improves performance in practice. * **Objective**: Our objective for IPMix is to improve model safety and adaptability to real-world complexity. While other P-level methods primarily focus on improving accuracy. * **Performance**: Compared to recent patch-level methods that utilize saliency information (e.g., PuzzleMix) to mix samples and labels, our approach achieves better performance without incurring additional computational complexity. We list the experimental results of IPMix and other existing methods on CIFAR-100 with WRN28-10 below. | | Classification (Error)↓ | Robustness (mCE)↓ | Adversaries (Error)↓ | Consistency (mFR)↓ | Calibration (RMS)↓ | |--|-|-|-|-|--| | AutoAugment | 17.7 $ \pm $ 0.11 |38.4 $ \pm $ 0.15 |97.8 $ \pm $ 0.22 |8 $ \pm $ 0.06 |7.9 $ \pm $ 0.06 | | RandAugment | 17.8 $ \pm $ 0.14| 41.5 $ \pm $ 0.13| 96.6 $ \pm $ 0.25| 8.6 $ \pm $ 0.10| 7.9 $ \pm $ 0.04| | TrivialAugment | 18 $ \pm $ 0.13| 35.4 $ \pm $ 0.21| 96.3 $ \pm $ 0.23| 7.3 $ \pm $ 0.07| 8.7 $ \pm $ 0.04 | | SaliencyMix | 19 $ \pm $ 0.14| 38.3 $ \pm $ 0.24| 96.7 $ \pm $ 0.21| 10.8 $ \pm $ 0.07| 7.1 $ \pm $ 0.07 | | PuzzleMix | 18.3 $ \pm $ 0.11| 37.9 $ \pm $ 0.21| 96.1 $ \pm $ 0.23| 10.5 $ \pm $ 0.04| 7.5 $ \pm $ 0.08 | | Co-Mixup | 18.1 $ \pm $ 0.19| 35.6 $ \pm $ 0.25| 95.6 $ \pm $ 0.21| 10.1 $ \pm $ 0.05| 7.7 $ \pm $ 0.04 | | Manifold Mixup | 18.6 $ \pm $ 0.21| 51.3 $ \pm $ 0.23| 93.4 $ \pm $ 0.17| 29.9 $ \pm $ 0.28| 10.2 $ \pm $ 0.09 | | IPMix | **17.4 $ \pm $ 0.17**| **26.6 $ \pm $ 0.19**| **91.3 $ \pm $ 0.21**| **4.2 $ \pm $ 0.11**| **6.4 $ \pm $ 0.07**| We hope this additional clarification sheds light on the issue. Regardless of the final decision, we genuinely appreciate your insights, which will help improve our work. [1] Hendrycks, Dan et al. “Unsolved Problems in ML Safety.” ArXiv abs/2109.13916 (2021): n. pag.
An extra supplement
We would like to share a critical observation from our experiments. While the combination of MixUp, CutMix, and AutoAugment is widely adopted as a default data augmentation strategy in many models, our findings suggest that merely combining multiple augmentation levels (pixel, patch, and image) might not always be optimal and can potentially degrade a model's safety performance. This observation is corroborated in our experiments with DeiT: when we **replaced** its default data augmentations (MixUp+CutMix+AutoAugment) with IPMix, the model's performance significantly improved. Hence, we believe that delving deeper into this area holds substantial value. | |Classification (Accuracy)↑| ImageNet-C (Error)↓| ImageNet-R (Error)↓| Calibration (RMS)↓| |--|-|-|-|-| DeiT |72 |57 |66.8 |13.3| DeiT + IPMix | **72.6 (+0.6)** |**50.4 (-6.6)** |**64.1 (-2.7)** |**10.5 (-2.8)**|
Raised my score.
Thanks for the hard work authors have put into the rebuttal. More of my concerns have been addressed. I am fine with accepting it now. I suggest the authors revise the abstract and introduction in the camera-ready to better reflect the novelty and contributions of the proposal, compared to the existing literature, and add the new results/discussions as they promised.
Thanks to Reviewer Bz96
Thank you for recognizing the effort we've put into our work and the rebuttal. We are truly grateful for your constructive feedback and guidance throughout this process. We will carefully consider your suggestions, revising the abstract and introduction to more distinctly emphasize the novelty and contributions of our proposal in the context of existing research. Moreover, we are committed to including the new results and discussions in the final manuscript as promised. Thank you once again for your invaluable insights that have significantly contributed to enhancing the quality of our paper.
Thanks for the Rebuttal
I appreciate the authors' efforts in rebuttal. I have read the response carefully. Most of my concerns are addressed, and I hope the authors can incorporate everything they promise into the final version, including the ImageNet study. I am raising my score from 5 (Borderline accept) to 6 (Weak accept). Good luck with your submission.
Thanks to Reviewer FKxz
Thank you very much for taking the time to revisit our submission and for your constructive feedback. We genuinely appreciate your acknowledgment of our efforts in the rebuttal. Please be assured that we are committed to incorporating all the promised improvements into the final version. Thanks again for your insightful responsiveness and for helping us improve our paper!
Reply to rebuttal
Thanks for the detailed reply from the authors and some of the concerns are addressed. However, the provided results on ImageNet are not very promising to the reviewer since the improvement is very limited and nearly all the results on ImageNet are obtained using tiny networks. As the authors explained in W3: the same method may exhibit different performances with different model capacities, the efficacy of IPMix on larger models and large-scale datasets remains a concern. After reading all the replies, the reviewer does feel that the efficacy of IPMix on large-scale datasets like ImageNet is not convincing enough and the technical contribution is also unclear. Therefore, the reviewer decides to keep the original rating.
Thanks for your reply
Thank you for the feedback on our submission. 1. We understand your concerns regarding the application of IPMix with larger models, and we'd like to address them as follows: * **Performance on Larger Models**: In our supplementary experiments with models like ConvNeXt-T and Swin-Tiny, which are **comparably sized** to the widely used ResNet-50 (ConvNeXt-T: 28M, Swin-Tiny: 28M, ResNet-50: 25.6M), IPMix has demonstrated significant performance gains. This indicates the **generalization ability** of our method and its potential applicability to larger architectures. **Experiments on ImageNet:** | |Classification (Accuracy)↑| ImageNet-C (Error)↓| ImageNet-R (Error)↓| Calibration (RMS)↓| |--|-|-|-|-| ConvNeXt-Tiny (28M) |81.5 | 43| 52.5| 7.2| ConvNeXt-Tiny + IPMix | **81.8 (+0.3)** | **36.2 (-6.8)** | **50.8 (-1.7)** | **7.1 (-0.1)** | Swin-Tiny (28M) |80.9 |47.1 |58.3 |7.3| Swin-Tiny + IPMix |**81.2(+0.3)** | **41.3(-5.8)** | **56.5(-1.7)** | **6.7(-0.6)** | * **Experiments on Small Models:** Due to computational and time constraints, we conducted many of our experiments on smaller models. However, it's worth noting that the improvements we achieved, especially on ImageNet, are **not minor**. We observed improvements across various safety metrics without compromising the model's accuracy, which is more meaningful from a practical perspective. * **Plug-and-Play Nature on ImageNet:** Our supplemental experiments have shown that IPMix exhibits strong **plug-and-play** when integrated with existing models, significantly enhancing their safety performance. We believe this attribute is **model-size agnostic**. Additionally, it's essential to recognize the efficiency of improving smaller models to match the performance of larger ones. For instance, a ResNet-18 enhanced with IPMix outperforms a vanilla ResNet-50 on ImageNet-C (ResNet-18+IPMix: **77.8** mCE, ResNet-50: 78.2 mCE), showcasing potential **resource savings**. * **Future Work on Bigger Architectures:** We are in the process of conducting experiments with more extensive models, including ResNet-50. We plan to integrate these results into the revised manuscript to address concerns about IPMix's effectiveness on larger models. 2. We apologize if the technical contribution appeared unclear. Our main contribution are follows: * **Improving Diversity and Complexity**: While data diversity and structural complexity are keys to model robustness, most previous methods to improve robustness focus on either generating diverse samples or increasing complexity. We are the **first** to propose a data augmentation method that mixes fractals with the original image at different levels and combines the strengths of these methods into a unified and coherent whole, to enhance the **overall performance**. * **Comprehensive Improving Safety Metrics:** Most methods target only accuracy or data shifts from corruption. In contrast, we aim to elevate performance across **multiple safety metrics**, which is more meaningful from a practical perspective. * **A Unique Paradigm:** Although some components of our approach have been proposed before, we believe that the paradigm of our entire method is **unprecedented**. For example, although the paper [1] introduced Random Pixels and Random Elements, the reported experimental results indicated that these methods achieved the lowest accuracy (4.3%↓ and 4.5%↓ by comparing with Mixup). We discover that mixing the original image and fractals using these two methods can **increase the robustness without sacrificing accuracy**. Our motivation is to combine these components that would typically affect the accuracy to showcase potential performance. We posit that this task is **non-trivial**, but rather holds significant value and can benefit future research. We will strive to make this clearer in our future revisions. We genuinely value your insights and will continue to enhance our work, keeping your feedback in mind. [1] Summers, Cecilia, and Michael J. Dinneen. "Improved mixed-example data augmentation." In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1262-1270. IEEE, 2019.
An extra supplement
We would like to share a critical observation from our experiments. While the combination of MixUp, CutMix, and AutoAugment is widely adopted as a default data augmentation strategy in many models, our findings suggest that merely combining multiple augmentation levels (pixel, patch, and image) might not always be optimal and can potentially degrade a model's safety performance. This observation is corroborated in our experiments with DeiT: when we **replaced** its default data augmentations (MixUp+CutMix+AutoAugment) with IPMix, the model's performance significantly improved. Hence, we believe that delving deeper into this area holds substantial value. | |Classification (Accuracy)↑| ImageNet-C (Error)↓| ImageNet-R (Error)↓| Calibration (RMS)↓| |--|-|-|-|-| DeiT |72 |57 |66.8 |13.3| DeiT + IPMix | **72.6 (+0.6)** |**50.4 (-6.6)** |**64.1 (-2.7)** |**10.5 (-2.8)**|
The results of IPMix on ImageNet
To better demonstrate the efficacy of IPMix on large-scale datasets such as ImageNet, especially with larger models, we've conducted additional experiments using ResNet-50. We trained the model for 180 epochs, adhering to the standard training scheme outlined in [1]. Due to computational and time limitations, we focused our experiments on IPMix, Vanilla, and AugMix (with results taken from the original paper). We firmly believe that these results provide a more compelling argument for the effectiveness of IPMix with bigger models on ImageNet. Furthermore, please be assured that the complete results will be included in the revised manuscript. **Experiments on ImageNet with ResNet-50:** | | Classification (Error)↓ | ImageNet-C (mCE)↓ | ImageNet-P (mFR)↓ |Calibration (RMS)↓ | |--|-|-|-|-| | Vanilla | 23.9 |80.6 |57.2 |5.6 | AugMix | 22.4 |68.4 |37.4 |4.5 | IPMix | **22.1** |**64.5** |**35.7** |**3.2** Please note that OpenReview will not allow us to address additional questions or concerns after tomorrow. Are there any further experiments you'd like us to perform, or do you have any other concerns? Once again, thank you for your invaluable insights and queries during the rebuttal phase. Your feedback has undeniably helped enhance our work. [1] Goyal, Priya et al. “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.” ArXiv abs/1706.02677 (2017): n. pag.
Thanks
I have read other reviews and the rebuttal. I would like to thank the authors for their effort. The rebuttal was helpful, thus I am increasing my rating to borderline accept. Please include the new results in the updated version of your manuscript (Full ImageNet results could be more useful for the robustness & safety researchers to truly assess the capability of this method).
Thanks to Reviewer jACH
Thank you for taking the time to re-evaluate our submission and for recognizing our efforts in the rebuttal. We genuinely appreciate your constructive feedback. Please be assured that we will include the new results in our revised manuscript. Once again, thank you for your insights and suggestions that have contributed to improving our paper.
Thank you for your thorough response. I have read your individual and global response and appreciated the detail, sourcing, and thoroughness. You have addressed my weaknesses and questions. Accordingly, I am raising my score from weak accept to accept with the expectation that the authors will make the revisions they indicated in my individual response.
Thanks to Reviewer jXDs
Thank you for reconsidering our paper and acknowledging our work during the rebuttal. We're glad to hear that our responses have addressed your concerns. Please rest assured that we will incorporate the revisions as indicated in our response to your comments. We greatly appreciate your feedback and support in improving our paper!
Decision
Accept (poster)