Recent advancements in masked image modeling (MIM) have made it a prevailing framework for self-supervised visual representation learning. The MIM pretrained models, like most deep neural network methods, remain vulnerable to adversarial attacks, limiting their practical application, and this issue has received little research attention. In this paper, we investigate how this powerful self-supervised learning paradigm can provide adversarial robustness to downstream classifiers. During the exploration, we find that noisy image modeling (NIM), a simple variant of MIM that adopts denoising as the pre-text task, reconstructs noisy images surprisingly well despite severe corruption. Motivated by this observation, we propose an adversarial defense method, referred to as De^3, by exploiting the pretrained decoder for denoising. Through De^3, NIM is able to enhance adversarial robustness beyond providing pretrained features. Furthermore, we incorporate a simple modification, sampling the noise scale hyperparameter from random distributions, and enable the defense to achieve a better and tunable trade-off between accuracy and robustness. Experimental results demonstrate that, in terms of adversarial robustness, NIM is superior to MIM thanks to its effective denoising capability. Moreover, the defense provided by NIM achieves performance on par with adversarial training while offering the extra tunability advantage. Source code and models are available at https://github.com/youzunzhi/NIM-AdvDef.
Paper
Similar papers
Peer review
Summary
This paper proposed a denoising pre-training framework NIM similar to MIM to improve adversarial defense ability. In particular, the input image is a noisy version of the original. The pre-training goal is to do denoising.
Strengths
This paper is easy to follow.
Weaknesses
* a) Novelty is limited. The idea proposed in this work is very similar to CIM [1]. Compared to MIM, this work changes the masking operation to add noise. * b) Performance is incremental with limited applicability. The proposed De3 as a pure self-supervised pre-training approach does not outperform MIM baselines (MAE and SimMIM). For downstream adversarial robustness tests, De3 shows improvement over MAE and SImMIM, which is not surprising since the pre-training goal is to denoise. * c) Datasets and backbones used in this work to validate performance are scarce. It is suggested that more datasets,downstream tasks, and backbones are evaluated. **References** [1] Fang, Yuxin, et al. "Corrupted image modeling for self-supervised visual pre-training." ICLR 2023.
Questions
See weakness.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
See weakness.
Further Discussion with Reviewer qarJ
Dear Reviewer qarJ, We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights. Please let us know if any aspect of our work remains unclear or if you have additional feedback. Thank you. Warm regards, Authors
Please take a look at authors' responses and other reviewers' comments
Dear Reviewer, Please take a look at authors' responses and other reviewers' comments, Thank you very much. BTW, for a solid review, it would be better to give more details to make a decesion. The current review seems to be short.
Summary
The authors introduce noise image modeling (NIM) as a self-supervised pretext task and demonstrate that their encoder-decoder architecture decreases the success of adversarial attacks by adding noise to a perturbed image and then denoising it. Their method is called $De^3$. Furthermore, the author conduct a major comparison effort to masked image modeling (MIM) and the capabilities against adversarial attacks.
Strengths
- This paper is well written and easy to understand. - "flooding out" adversarial attacks is an interesting idea. - The method utilizes the usually unused decoder for "cleaning" adversarial images. - $De^3$, as an adversarial prevention method, can be applied without generating much computation overhead during inference or training. - NIM with $De^3$ can use a dynamic trade-off between clean accuracy and the vulnerability to adversarial images. - On-par results with MIM while doing adversarial training.
Weaknesses
1. Table 1/Figure 3: - The "flooding out effect" for very large sigma (140) seems not reasonable, since it corrupts the whole image. How strong is the flooding in comparison to the adversarial perturbation? I think it is necessary to show the scale difference of (i) the adversarial attacks and (ii) the added noise. - I am a little bit surprised by the denoising performance with high noise-levels (see Figure 3 (f) - sigma 140): the reconstruction seems "too good to be true", where it even recovers the smallest features. This seems odd despite other efforts [1]. I think it needs further extensive investigation: (i) when does the model break in terms of reconstruction quality? What happens when you use an even larger value for sigma? (ii) when does the adversarial defense fail when a high-noise level (sigma 100, 150, 200) is used? 2. Figure 2 (a) needs captions for the image rows/columns to understand it better. 3. Table 1: Inconsistent use of bold highlighting? 4. Table 1: I am not sure if this is a fair comparison, since NIM utilizes its built-in adversarial prevention while MAE/SimMIM (MIM) has none (gamma is not implicitly designed for dealing with adversarial images). 5. line 264-268: From the readers perspective, it seems that the authors tried to achieve state-of-the-art with $De^3$ but failed, so they redirect their writing to fit a different goal. I suggest, the authors omit the sentences about not wanting to achieve SOTA. 6. Figure 5: I suggest to use more symbols for different sigmas. Differentiate between fixed/random sigma and MAE baseline with triangle and then use square/diamond/circle. 7. Figure 5: Ordering of the entries in the legend can be improved References: [1] Mahdaoui, Assia El, Abdeldjalil Ouahabi, and Mohamed Said Moulay. "Image denoising using a compressive sensing approach based on regularization constraints." Sensors 22.6 (2022): 2199.
Questions
Do you think that there is a way to construct strong adversarial examples if the attacker knows that the NIM architecture is used, despite the noise randomness?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Limitations are not discussed in the paper.
Further Discussion with Reviewer XF5y
Dear Reviewer XF5y, We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights. Please let us know if any aspect of our work remains unclear or if you have additional feedback. Thank you. Warm regards, Authors
Thanks for your responses and the clarifications. My concerns were addressed in an appropriate way. Overall, NIM with De^3 seem to be interesting, novel and competitive with existing related methods. My main issue was the reconstruction quality for high sigmas which apparently seems to work. Hence, I am increasing my rating to 7.
Summary
The paper presents a straightforward approach for defending against adversarial attacks while incorporating pre-trained feature learning through the utilization of noisy images. Inspired by Masked Image Modeling (MIM), the paper replaces the masking pretext task by introducing a substantial amount of noise into the image. Subsequently, a transformer-based encoder/decoder is trained to reconstruct the original image, and the encoder is fine-tuned to acquire features that are invariant to noise by comparing the encoded features of the original image input with those of the reconstructed images. The defense mechanism involves retaining the decoder for the downstream task, employing an [encoder → decoder → encoder] mechanism to obtain features that can be further fine-tuned for the downstream task. The authors perform experiments on ImageNet-1K using the ViT-Base backbone and evaluate the proposed method against several existing adversarial attacks, including $l_{\infty}$ bound attacks, FGSM, PGD, and AutoAttack (AA). The results demonstrate that the proposed method exhibits greater robustness compared to baseline pre-trained methods like MAE or SIMM.
Strengths
The finding of the paper is interesting, which suggests that the adversarial attacks present as a kind of noise, and it makes sense that denoising works to some extent. This aspect might open up possibilities for combining existing works to further enhance downstream accuracy and adversarial defense. The paper writing is easy to follow and the proposed method is simple and comprehensive.
Weaknesses
1. It would be beneficial to have a more extensive comparison by including well-known self-supervised features like SimCLR, MOCO, DINO, and others, to assess the robustness of the features learned by these methods in comparison to the proposed method. 2. To enhance the paper, it would be valuable if the authors conducted experiments on another dataset and included more baseline methods to gain insights into the generalizability of the proposed method. 3. It would be valuable to compare the proposed method with existing adversarial defense approaches, such as GAN-based methods. 4. Despite some improvements, the defense results are not particularly impressive and do not appear to completely eliminate the attacks. This suggests that the attacks might involve more than just noise. 5. It should be noted that using additional decoders could be a drawback depending on the applications, as it significantly increases memory usage during inference. The approach seems novel, and I find the current version of the paper is fine unless there are serious concerns / flaws from other reviewer’s feedback. However, I would recommend conducting additional experiments to further showcase the generalizability of the proposed method.
Questions
I wonder if the authors have considered keeping the original MAE approach and combining it with the noisy modelling technique to explore any potential further improvements? See questions above also.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
N/A
Further Discussion with Reviewer ovhG
Dear Reviewer ovhG, We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights. Please let us know if any aspect of our work remains unclear or if you have additional feedback. Thank you. Warm regards, Authors
Summary
This paper proposes a novel adversarial defense method $De^3$ to utilize the strong denoising ability of NIM models. The proposed method first adds some Gaussian noise to the adversarial samples and then tries to reconstruct the original images. Experiments show the advantage of NIM over MIM in terms of adversarial robustness.
Strengths
1. The idea of NIM is interesting. The proposed method obtains the adversarial defense as well as provides pretrained features by a simple yet effective modification on masked image modeling. 2. The idea of De3 is instructive to enhance adversarial defense through reconstructing clean images from intensely noisy images. 3. The experiments demonstrate that the proposed method can achieve comparable defense performance with adversarial training. 4. This paper is well-written and easy to follow.
Weaknesses
1. Although De3 can enhance adversarial defense, it increases computational cost during inference time. Note that there exists two encoders and one decoder in De3, can we distill another encoder from them and use only it during inference time? 2. This paper compares NIM against MIM in terms of accuracy and adversarial robustness. It seems that MIM achieves better accuracy on clean images. I wonder can we combine them together and obtain a better trade-off between accuracy and robustness?
Questions
I wonder the difference of robustness against other types of attacks, like black-box ones.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
4 excellent
Limitations
Limitations are adequately discussed in the supplemental materials.
Summary
This paper proposes a method called Noisy Image Modeling (NIM) as self-supervised learning to improve the adversarial robustness of pretrained features. NIM uses denoising as a pretext task and is effective in reconstructing noisy images for representation learning. In addition, the authors propose a defense technique called De^3 that leverages the denoising capability of NIM to enhance robustness against adversarial attacks. Experimental results show that NIM with De3 defense outperforms Masked Image Modeling (MIM) in terms of adversarial robustness while maintaining competitive performance on clean data. The paper concludes by highlighting the potential of NIM and other variants of MIM for generative visual pretraining.
Strengths
1. Different from the popular MIM, NIM improves the pretrained features by resisting adversarial attacks. The experiment verfies the effectiveness of the proposed method. 2. In addition to the representation learning method, the paper proposes a defense technique called De3 that utilizes the denoising capability of NIM to enhance robustness against adversarial attacks. 3. The paper compares the performance of NIM with the proposed De3 defense to Masked Image Modeling (MIM) and other adversarial training methods. Experimental results show that NIM with De3 defense outperforms MIM in terms of adversarial robustness while maintaining competitive performance on clean data. 4. The paper introduces a modification to NIM that allows for a tunable trade-off between accuracy and robustness.
Weaknesses
1. The paper does not thoroughly analyze the computational cost of the proposed method. Adversarial training methods, such as the one proposed by Wang et al. [34], are known to be computationally expensive. It would have been valuable to compare the computational cost of NIM with other defense techniques. 2. The paper does not extensively explore the impact of hyperparameters on the performance of the proposed method. For example, the authors mention that the trade-off between clean accuracy and robustness can be adjusted by varying the noise level hyperparameter, but they do not provide a detailed analysis of the optimal values or the sensitivity of the method to different hyperparameter settings. 3. The paper does not analyze the transferability of adversarial examples between different models. It would have been valuable to investigate whether the robustness achieved by NIM with De^3 defense can generalize to other models or if it is specific to the pretrained model used in the experiments [1].
Questions
1. Computational cost. 2. Parameter Sensitivity Analysis. 3. The generalization of the proposed method.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes
I thank the authors for their response. However, I still have the following concerns: * CIM is a very representative work in the MIM domain. Since it is very similar to the method proposed by the authors, at least it should be cited. Moreover, different research purpose does not distinguish the two works. For instance, applying the idea of MAE to audio pre-training is an application but not an innovation. * The pre-training objective of CIM is to denoise the corrupted input image, which is exactly the same idea proposed by the authors in this work. However, although the architectural designs and corruption approaches might be different, the underlying idea is the same. * Adding Gaussian noise to the input image and asking the model to denoise during pre-training improves the model's robustness towards noise in general. * More results on datasets with various scales and types are required to draw solid conclusions. For instance, IN-21k, iNaturalist, COCO etc. With these concerns, I will stand by my rating.
Further discussion with Reviewer qarJ (Part 1/2)
> CIM is a very representative work in the MIM domain. Since it is very similar to the method proposed by the authors, at least it should be cited. > While we acknowledge that CIM is a representative MIM variant, we would like to emphasize again that it has little similarity with our proposed method. In CIM, the authors proposed to improve MIM by using an auxiliary trainable BEiT to degrade the images. In contrast, our paper not only alters the degradation technique but also presents a novel framework that leverages the pretrained model to defend adversarial attacks beyond merely using pretrained representations. In Section 2.1 (lines 78-85), we've already distinguished our work from other research employing different degradation methods and talked about their inspiration for our exploration into NIM. In our final paper, we will cite CIM and include the discussion here for a more comprehensive related works section. > Moreover, different research purpose does not distinguish the two works. For instance, applying the idea of MAE to audio pre-training is an application but not an innovation. > First, we would like to argue that different research purposes usually do distinguish two works. Taking the example of Vision Transformer [1], the authors claimed that they “have explored the direct application of Transformers to image recognition” (see Conclusions in [1]). Yet the paper has become one of the most important works in machine vision. Another good example is “a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos” [2], which was accepted by this venue last year and became influential in its domain. Moreover, our paper is clearly not a simple application of CIM (or any other MIM variants) to the field of adversarial robustness. Regarding the example proposed by the reviewer, it is imaginable that a completely direct application of MAE to audio could result in unexciting work, but in the case of our work, the vast disparity between the domains of adversarial robustness and image recognition makes it unlikely to directly apply CIM or its concepts to adversarial robustness. In light of these factors, we insist that our work is very different from CIM. > The pre-training objective of CIM is to denoise the corrupted input image, which is exactly the same idea proposed by the authors in this work. However, although the architectural designs and corruption approaches might be different, the underlying idea is the same. > A closer look at CIM reveals that its objective diverges from ours. CIM studied two pre-training objectives, one of which is to recover the images corrupted by a small trainable BEiT generator where a pre-trained frozen image tokenizer encoder and decoder are involved, and the other is a discriminative one. Notably, terms like "denoise" or "noise" aren't explicitly mentioned within the CIM paper. In sum, the objective of CIM is much different than “denoise the corrupted input image”. It's worth noting that even if NIM shares some similarities with CIM, equating the two as having the "same idea" is an oversimplification. If one were to perceive CIM and NIM as identical in conception, then by extension, many MIM variants, like MFM [4] and even CIM, would merely be replicas of the original MIM concept and, therefore, lack novelty. Even MIM can be seen as not novel because the idea is the “same” as MLM. However, this isn't the prevailing perspective in our scientific community, and rightfully so. As Professor Michael Black aptly articulated [3], “Taking an existing network and replacing one thing is better science than concocting a whole new network just to make it look more complex.” Furthermore, it's imperative to highlight that our work isn't solely confined to NIM. We further propose $De^3$, a novel framework that effectively leverages the pretraining NIM models for providing adversarial defense to downstream models. This is a significant contribution of our work which is recognized by other reviewers as a novel and interesting method. --- [1] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. [2] Feichtenhofer et al. Masked Autoencoders As Spatiotemporal Learners. NeurIPS 2022. [3] Black. Novelty in Science: A guide for reviewers. https://perceiving-systems.blog/en/post/novelty-in-science.
Further discussion with Reviewer qarJ (Part 2/2)
> Adding Gaussian noise to the input image and asking the model to denoise during pre-training improves the model's robustness towards noise in general. > We'd like to reference Michael’s blog [3] again where he talks about the relationship between novelty and surprise: ”The novelty, however, must be evaluated *before* the idea existed… If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea.” To the best of our knowledge, the idea of using the ‘adding-noise-then-denoising’ pretraining to enhance the model’s robustness against adversarial noise has not been proposed before. Therefore, we argue that the idea of our work is novel even it may seem obvious in hindsight. > More results on datasets with various scales and types are required to draw solid conclusions. For instance, IN-21k, iNaturalist, COCO etc. > Following your suggestion, we have shown in the rebuttal that our method is effective on CIFAR-10 besides ImageNet. We humbly disagree that it is necessary to experiment on more datasets to prove the solidity of a method. Previously published studies often substantiate their claims based on results from a handful of datasets. For example, the recent work [5] we compared in Section 5.3, shows the empirical results on CIFAR-10 and Imagenette (a subset of 10 classes from ImageNet-1K). Another related work [6] conducts experiments solely on ImageNet. Similarly, an earlier work on adversarial robustness [7] also only used ImageNet to show the empirical results. Given these precedents, we believe that our choice of ImageNet and CIFAR-10 - both of which are well-regarded and commonly used datasets in the community - provides a sufficient foundation to support our conclusions. --- [3] Black. Novelty in Science: A guide for reviewers. https://perceiving-systems.blog/en/post/novelty-in-science. [4] Xie et al. Masked Frequency Modeling for Self-Supervised Visual Pre-Training. ICLR 2023. [5] Mo et al. When adversarial training meets vision transformers: Recipes from training to architecture. NeurIPS 2022. [6] Kong and Zhang. Understanding Masked Image Modeling via Learning Occlusion Invariant Feature. CVPR 2023. [7] Xie et al. Feature Denoising for Improving Adversarial Robustness. CVPR 2019.
* Similarity. The similarity between CIM and your proposed method does not magically disappear because you claim or emphasize that there is no similarity. For any self-supervised pre-training, the pre-text task is a crucial factor. Since both CIM and the proposed NIM denoise, the authors should at least compare and state the similarities and differences in their paper. However, they failed to do so. * Comparison. The authors compared with MIM and simMIM. Yet the authors claimed that MIM, simMIM and CIM all seek to learn s to learn better pretrained visual representations while NIM generative pretraining paradigm can provide adversarial robustness. Why compare with and MIM and simMIM in the first place but not some other pre-training methods designed for adversarial robustness? * Novelty. I mentioned CIM only because the authors compare with MIM and simMIM. Besides, NIM also share with same idea with denoising autoencoder [1]. The authors are suggested to compare with [1] using same encoder as pre-training. * More experiments. The authors claimed to propose a pretraining paradigm. A generic paradigm should work with various backbones and datasets. The authors compared with pre-training methods like MIM and simMIM. Then they should at least have the same scale of experiments. Meaning they should at least use the same number of backbones as MIM and simMIM. [6] mentioned by the authors are explainability work but not NIM. It is acceptable for [6] to have less experiments since its contribution is explainability. However, NIM requires more experiments to show effectiveness. [7] is a paper not in the self-pre-training domain. The standards from different domains are different. Also, [7] is an outdated paper. In recent years, more solid experiments are required by the community. * BTW, it is very improper for the authors to quote a blog lecturing the reviewer on how to be a reviewer. I feel very offended by such action. Please keep the discussion professional. Referecenes: [1] Vincent, Pascal, et al. "Extracting and composing robust features with denoising autoencoders." Proceedings of the 25th international conference on Machine learning. 2008.
Further discussion with Reviewer qarJ (Part 1/2)
> Similarity. The similarity between CIM and your proposed method does not magically disappear because you claim or emphasize that there is no similarity. For any self-supervised pre-training, the pre-text task is a crucial factor. Since both CIM and the proposed NIM denoise, the authors should at least compare and state the similarities and differences in their paper. However, they failed to do so. > We appreciate your suggestion and would like to state the similarities and differences between NIM and CIM here and will include the comparison in our final paper. Similarities: - Conceptually, both CIM and NIM are MIM variants, i.e., generative pretraining paradigms that replace the masking operation in MIM with some other degradation method. Differences: - The degradation methods in CIM and NIM are different. - In CIM, the input images are corrupted by a generator, which consists of a pretrained frozen image tokenizer consisting of a paired encoder and decoder, and a small trainable BEiT. In particular, tokens at masked positions that are sampled according to the small BEiT output distribution together with the golden tokens that are directly produced by the image tokenizer encoder at non-masked positions constitute the input for the image tokenizer decoder. And the decoder then maps the visual tokens to corrupted images. - In NIM, the input images are corrupted by Gaussian noises. - The pre-text tasks of CIM and NIM are different. - In CIM, the authors study two pre-text tasks, ResPix and RevDet. ResPix is a generative task where the enhancer predicts the uncorrupted pixel value for all positions given the corrupted images, and RevDet is a discriminative task where the enhancer determines whether each visual token is replaced by a generator sample or not. - In NIM, the pre-text task is to denoise the added Gaussian noise. - The hyperparameter that controls the difficulty of the pre-text task is different. - In CIM, the difficulty of the pre-text task mainly depends on the masking ratio, which is set to 50% globally in a pre-determined way. - In NIM, the difficulty of the pre-text task depends on the $\sigma$ of the added Gaussian noise, which is sampled from a $\Gamma$ distribution. - CIM and NIM are used in different ways in the two papers. - CIM is only used for providing pretrained visual representations. Only the enhancer is used for downstream fine-tuning and the generator is thrown away. - NIM is used not only for providing pretrained features but also for adversarial defense. Under our proposed novel framework, $De^3$, both encoder and decoder are utilized for enhancing the adversarial robustness of downstream models. > Comparison. The authors compared with MIM and simMIM. Yet the authors claimed that MIM, simMIM and CIM all seek to learn s to learn better pretrained visual representations while NIM generative pretraining paradigm can provide adversarial robustness. Why compare with and MIM and simMIM in the first place but not some other pre-training methods designed for adversarial robustness? > We compare with representative MIM pretraing methods because the goal of this work is to show that NIM can achieve a stronger and tunable accuracy-robustness trade-off compared to MIM. We have also compared our NIM+$De^3 method with a recent adversarial training method in Section 5.3 to help understand the effectiveness of our method. > Novelty. I mentioned CIM only because the authors compare with MIM and simMIM. Besides, NIM also share with same idea with denoising autoencoder [1]. The authors are suggested to compare with [1] using same encoder as pre-training. > While we hope to show the experimental results comparison with CIM as we did with MAE and SimMIM, we cannot find any code implementing CIM, official or unofficial, and it is very challenging to reimplement CIM within this short time of the discussion period. As we have discussed in lines 127-130, the term “denoising” in DAE [1] means recovering masked pixels, instead of removing added Gaussian noise as in NIM. Following your suggestion, here we show a performance comparison with DAE using the same encoder. Since the time is limited, we conduct the experiments on CIFAR-10, following again the implementation of [a], and only pretrained DAE and NIM-MAE for 200 epochs with 20 epochs for warmup. The results are as follows: | Model | DAE | DAE | NIM-MAE | NIM-MAE | | --- | --- | --- | --- | --- | | $De^3$ | None | $\gamma$=0.75 | $\sigma$=40 | $\sigma$=70 | | Clean | 71.44 | 52.89 | 76.44 | 71.20 | | FGSM | 5.14 | 26.03 | 25.98 | 35.78 | | PGD-10 | 0.05 | 8.08 | 15.15 | 26.88 | | AA | 0.00 | 5.06 | 16.17 | 27.62 | It is shown that NIM outperforms DAE in both clean accuracy and adversarial robustness. --- [a] https://github.com/IcarusWizard/MAE.
Further discussion with Reviewer qarJ (Part 2/2)
> More experiments. The authors claimed to propose a pretraining paradigm. A generic paradigm should work with various backbones and datasets. The authors compared with pre-training methods like MIM and simMIM. Then they should at least have the same scale of experiments. Meaning they should at least use the same number of backbones as MIM and simMIM. [6] mentioned by the authors are explainability work but not NIM. It is acceptable for [6] to have less experiments since its contribution is explainability. However, NIM requires more experiments to show effectiveness. [7] is a paper not in the self-pre-training domain. The standards from different domains are different. Also, [7] is an outdated paper. In recent years, more solid experiments are required by the community. > In MAE, the authors conduct experiments with ViT-B, ViT-L, and ViT-H as backbones, and we have also shown that our method is effective on these backbones. Besides, we have followed your suggestion and shown that our method is effective on CIFAR-10 besides ImageNet. While [6] performed some theoretical analysis, its main contributions are based on experimental results, so sufficient experiments are needed to prove their findings solid. Also, we hope the reviewer may consider the explanation in our rebuttal that performing adversarial attacks for other tasks than classification is beyond this paper’s scope. Overall, we would like to clarify again that the purpose and novelty of our paper are not to propose a new pretraining paradigm, but to show the superiority of NIM over MIM in terms of adversarial robustness using the proposed defense framework, and we believe that our experimental results are sufficient to prove this point. > BTW, it is very improper for the authors to quote a blog lecturing the reviewer on how to be a reviewer. I feel very offended by such action. Please keep the discussion professional. > We deeply apologize for the unintended offense caused by referencing the blog. It was never our intention to lecture or undermine your expertise. Our intent was solely to share an understanding of novelty and believe that quotations from influential scholars may make such an understanding more convincing. We notice that discussions on OpenReview sometimes reference external articles or blogs. For example, in [b, c], the same blog was referenced by the authors. Nevertheless, we recognize the importance of ensuring our discussion remains respectful and professional at all times. Once again, we sincerely regret the oversight and appreciate your feedback. --- [b] https://openreview.net/forum?id=2OdAggzzF3z¬eId=0X9_fcsILKT [c] https://openreview.net/forum?id=nxl-IjnDCRo¬eId=JqTRaHS43ww
I appreciate the authors for dealing with my concerns and including more experiments. The additional results make me more convinced.
Decision
Accept (poster)