We propose a conceptually simple and lightweight framework for improving the robustness of vision models through the combination of knowledge distillation and data augmentation. We address the conjecture that larger models do not make for better teachers by showing strong gains in out-of-distribution robustness when distilling from pretrained foundation models. Following this finding, we propose Discrete Adversarial Distillation (DAD), which leverages a robust teacher to generate adversarial examples and a VQGAN to discretize them, creating more informative samples than standard data augmentation techniques. We provide a theoretical framework for the use of a robust teacher in the knowledge distillation with data augmentation setting and demonstrate strong gains in out-of-distribution robustness and clean accuracy across different student architectures. Notably, our method adds minor computational overhead compared to similar techniques and can be easily combined with other data augmentations for further improvements.
Paper
Similar papers
Peer review
Summary
The authors propose a framework that improves the image classification model's robustness by distilling CLIP models and augmenting adversarial learning with pre-trained generative models. The authors follow classical adversarial learning to generate perturbed examples and then input the examples to VQ-GAN. Lastly, they distill a CLIP model into a smaller network with both normal and augmented adversarial examples. While I have some concerns about the results, extensive experiments verify the effectiveness of the proposed method in many settings.
Strengths
1. The extensive experiments cover various settings, such as different architectures for teacher and student models and variations of ImageNet. 2. The authors provide some theory explaining that a more diverse distribution could be helpful for distillation.
Weaknesses
1. The research problem is unclear. In L37, the authors claim to "train small robust models with a large-scale model as a teacher, but without access to the teacher's training data." However, they focus more on out-of-distribution robustness in the later parts and the experiments. 2. The proposed method needs to be better justified. In L118, the authors suddenly introduce generative models to augment adversarial examples and claim that it helps generalizability without any reasoning. If possible, the authors should provide more motivation and insights into it and conduct some experiments. 3. Many notations are not defined. For example, the definition of $\theta(x, x\prime)$ is missing. What exactly does $Q(x\prime)$ work? 4. The novelty is limited. The main contribution seems solely to leverage VQ-GAN to enhance distillation. The reason for choosing VQ-GAN is also missing. 5. The research scope is limited. The authors claim in L45 that distilling large-scale models into smaller models improves adversarial robustness. However, the experiments only consider CLIP models.
Questions
1. The performance could be better, even though it outperforms prior work. In Table 1, neither the baselines nor the proposed method achieves the CLIP baseline performance. Maybe one should look into cross-modality instead of augmenting query datasets for distillation. 2. What does it mean by *DAD (no distill)*? I recommend the authors refer the readers to the corresponding loss terms. 3. The proposed method is similar to leveraging more data to perform distillation (Eq. 8). The author may like to explore using different kinds of data (e.g., ImageNet + CIFAR-10 or ImageNet + ImageNet with augmentation), which could even outline the contribution of the proposed method.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
1. The authors only consider distilling CLIP models.
Summary
This paper proposes a simple and lightweight framework for improving the robustness of vision models through knowledge distillation. This paper applies pre-trained teacher model to generate adversarial examples and apply VQGAN as a data augmentation method to generate more informative adversarial samples. This paper provides a theoretical framework for applying robust teacher in the knowledge distillation with data augmentation.
Strengths
+The paper applies a large pre trained model based on knowledge distillation to enhance the model's out-of-distribution robustness with data augmentation, this idea is novel. +The paper provides a theoretical analysis of how to best distill robustness from a robust teacher trained on a large-scale dataset. +The experiment of the paper can prove the effectiveness of the method.
Weaknesses
-The motivation for applying VQGAN is not so clear. The article claims that VQGAN can discretize adversarial examples, but can other data augmentation technology such as common data augmentation methods (translation, cropping), other variants of GAN, or Stable Diffusion achieve better performance? This part seems to be a direct application, and I think a discussion of the necessity should be provided. -Although this paper has a pretty performance of out-of-distribution robustness, I wonder about the performance of adversarial robustness (such as PGD or AutoAttack). -The frame diagram is somewhat rough and crude (this will not affect the final rating, but I hope the author can make some modifications).
Questions
see the weakness
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
see the weakness
Follow up on adversarial robustness ablation
The reviewer asked about the adversarial robustness of models trained with DAD. We previously offered a simple result where there is improved small improvement in robustness against FGSM for ResNet50 and now have results on PGD and AutoAttack and on ViT-B. Below is the updated table with a more comprehensive evaluation. | Method | FGSM | PGD | AutoAttack | | --- | --- | --- | --- | | ResNet50 | 23.5 | 1.0 | 0.0 | | ResNet50 DAT | 33.0 | 5.9 | 0.0 | | ResNet50 DAD | 43.5 | 12.6 | 0.0 | | ViT-B | 49.4 | 24.7 | 0.0 | | ViT-B - DAT | 64.9 | 26.2 | 0.0 | | ViT-B - DAD | 47.2 | 25.0 | 0.0 | There is a small improvement in adversarial robustness for simpler attacks, but neither DAT or DAD is robust to AutoAttack as the VQ-GAN discretizes the perturbation. We observe that DAT is stronger than DAD for ViT-B. Unlike out-of-distribution robustness, since adversarial robustness is based on perturbations generated with gradients from the base model, DAT models are trained on images closer to these perturbations than DAD models (which were trained on perturbations generated with CLIP gradients). However, for ResNet50, DAD is better even for adversarial robustness as distillation is able to help smaller capacity models learn discrete adversarial examples, like we observe in Table 1. We hope this result is helpful and better answers the reviewer's question. We would be happy to answer any further questions.
Summary
The paper introduces a new method to improve the robustness of vision models through knowledge distillation by leveraging a large-scale pre-trained teacher model (CLIP) and a VQ-GAN discretizer. The paper does a great job motivating the method and arguing for knowledge distillation when tackling out of distribution robustness. It also includes extensive theoretical justifications. And finally, the paper introduces and evaluates a novel distillation objective ("DAD"). In the proposed method ("DAD") the student model is trained to minimize a loss made up of a standard cross-entropy loss, a KL to the teacher's predictions (i.e. standard knowledge distillation) and an additional KL between the teacher and student's predictions on additional (teacher-adversarial) data points (which is novel). The paper also includes a reasonable evaluation section for the method with overall positive results, while highlighting some considerable performance improvements on natural distribution shifts over baselines.
Strengths
The paper is relatively well written and easy to follow. The paper introduces a relatively novel and interesting knowledge distillation loss which should have increasingly wider applicability and grow in significance as more and more of the ML world is moving towards leveraging pre-trained foundation models.
Weaknesses
- Missing paragraph detailing DAT: since a large proportion of the strongest results seem to be relying on combining DAD with DAT. The paper should also include a description of DAT's loss and the "DAT + DAD" loss written out (in the appendix if no space available in the main body). - The paper mentions the method has "negligible computational overhead" -- I strongly disagree. Computing adversarial examples for a CLIP model that also passes them through VQGAN is not negligible. One can say that the computation cost is amortized though -- as you only have to do it once and then read it from disk. - Missing "Avg" column in Table 2 and 3, and best results are not shown in bold in Table 2. - Loss is unclear (Eq 9) -- please expand and clarify this to show which variables are being optimized over, at which stage, which are the constraints exactly, and specify the losses (what is l_1 and l_2 precisely - these are only mentioned in passing in line 125-126 being "typically"/"usually" CE and KL); and also please use or remove "n", as currently it is not used to index anything. - Missing references, e.g., on [1] (on worst case risk over a population of distributions) and [2] (for a related formulation of a similar set of assumptions and similar theoretical conclusion): [1] Arjovsky et al., Invariant Risk Minimization. [2] Calian et al., Defending against image corruptions through adversarial augmentations. - Figure 1 does not seem to be faithful depiction of the method as described in the paper, so it should ideally be improved significantly (the caption could also give more details towards explaining the method).
Questions
- At a high level, looking at equation (9) of the full model loss, I'm trying to understand how each component affects downstream robustness. So, I'd like to able to map the contribution of each of the two "l_2" losses and the way in which the adversarial example is computed to specific results. This seems like a pretty straightforward and important ablation. Table 3 and Table 5 present just a slice of this ablation as far as I can tell. Could you please clarify how one can reconstitute this ablation from the results presented in the paper? - The adversarial examples could also not be computed but rather perturbations could be sampled instead (while still using the VQGAN), or random data augmentation methods (or just an identity function used, i.e. which would correspond to just dropping the last l_2 loss in Eq 9). (From Table 6 it seems that more iterations actually results in worse performance across many metrics.) Have you experimented with this? - In Table 1, does the row "DAT + DAD (Ours)" refer to training with DAT & DAD on top of AugReg-ViT[55]? If yes, please mark it with a leading "+" for clarity. - In line 303, the paper states "These adversarial examples are generated with DAD" -- was "DAT" intended?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors adequately addressed the method's limitations.
Summary
This paper introduces a novel approach called discrete adversarial distillation (DAD) to train small, robust models using large-scale models as teachers. The authors establish the knowledge distillation framework for out-of-distribution robustness and provide a theoretical framework for utilizing large-scale models as teachers. DAD achieves state-of-the-art performance on natural distribution shifts, surpassing adversarial training and traditional distillation techniques.
Strengths
Overall, this paper is also well-written and clearly structured; the authors provide both theoretical analysis and empirical experiments to verify their approach. The reviewer would like to list some strengths of the work. Firstly, it extends the knowledge distillation framework to the domain of out-of-distribution robustness, opening up new possibilities for training small, robust models that can generalize effectively across diverse populations and environments. Secondly, this paper highlights the advantages of leveraging large-scale models as teachers, providing a theoretical framework to support this approach. This understanding can inform the development and utilization of robust models in practical applications. Lastly, the experimental results showcase state-of-the-art performance on natural distribution shifts, surpassing existing techniques. This signifies the potential of the proposed DAD method in addressing the challenges of robustness and generalization in ML models.
Weaknesses
The reviewer found it hard to understand the main idea at first glance, the paper could improve its clarity in figures, as well as in presenting the assumptions and theoretical analysis, and its connection to the proposed method. Explicitly linking the theoretical framework to the practical implementation of the DAD method would help readers grasp the underlying principles and motivations behind the proposed approach more effectively.
Questions
Please refer to the weakness. In addition, why the authors did not provide the code for review but answered the *Reproducibility* as "yes"? Please provide the code during the rebuttal period, using Anonymous GitHub is a viable option. Also, the authors claim that the proposed only brings negligible computational overhead, but the reviewer could not find the part that addresses computing resources; what exactly are the computational costs for the proposed framework?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
N/A
Summary
This paper presents a knowledge distillation framework for vision model, by leveraging a teacher model (CLIP). In the setup, a discretizer (VQGAN) model is introduced to the adversarial examples from the teacher model. Adversarial training (AT) objective is adapted in the knowledge distill setting. The proposed discrete adversarial distillation (DAD) demonstrate improved performance in out-of-distribution tasks, comparing to other AT approaches.
Strengths
This paper present an interesting approach for knowledge distillation for large scale vision model. The experiment is thoroughly conducted with various of SOTA approaches compared, and comprehensive ablation study. The writing is easy to follow and the idea is well established.
Weaknesses
1. One main concern is on comparing the proposed method with DAT. In DAT, the idea of discretizer is introduced and VQGAN was also considered in their framework. In their paper, various tasks were evaluated, including image classification, self-supervised learning, and object detection. To me, DAD seems to be applying the similar methodology, but on a OOD setup. 2. In the experiment, it seems like DAT has achieved competitive results in ImageNet-1K, and even better results in ImageNet-21K. What is the key advantage of DAD over DAT?
Questions
1. How come DAD+DAT performs worse than DAT in some cases? 2. Are there any qualitative results? what are the drawbacks from previous model(s) that are better addressed by the newly proposed DAD? 3. Have you try training the teacher model in parallel with the student model, not just provide adversarial samples from offline manner?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Please see weakness.
Follow up on data augmentation ablation
The reviewer asked about sampling from the VQGAN without computing the gradient. As promised in the initial response we provide an additional result which was not finished then on sampling from the VQGAN without using gradients. The results are significantly worse than DAD, indicating the need to use gradients to discover diverse samples. This is also supported by our theoretical analysis that indicates more diverse adversarial examples are better for robustness. Higher in-distribution performance also suggests the samples are less diverse. | | ImageNet | V2 | R | Sketch | A | Avg | | --- | --- | --- | --- | --- | --- | --- | | VQGAN - Sample | 80.9 | 70.1 | 49.3 | 34.9 | 24.0 | 51.8 | | VQGAN - Grad | 79.6 | 69.9 | 65.1 | 46.1 | 31.8 | 69.5 | We hope this result is helpful and improves our response to the reviewer's question.
Response to the Authors
I would like to thank the authors for the detailed response. After reading the response and carefully re-reading the paper, my concern about the research problem in this paper is partially addressed. However, the presentation of the paper remains a main issue. Many descriptions in this paper are unclear or even confusing. I would like to clarify my understanding below before adjusting my score. 1. In L21-26, it looks like the authors plan to work on improving the in-distribution accuracy under adversarial learning. But, it turns out to investigate data augmentation with the *zero-shot generalization* of foundation models (L33). 2. L36-38, again, is confusing. As mentioned in my previous comment, it gives the impression that the goal is solely to distill large models into smaller ones. 3. Is L40-L46 a finding from the authors or cited from other works? Does this only happen to CLIP? 4. Following (3), Table 7 in the appendix is important evidence to motivate the proposed method, where normal augmented data are unsuitable for distilling CLIP models. It is unclear why the authors leave them in the appendix. 5. The work's main contribution is to leverage discretizers, i.e., VQ-GAN, and the idea is highly built upon the prior work DAT. However, relations, detailed comparisons, or motivations are totally missing in this work, making the readers hard to judge the contribution. The operation of VQ-VAE is also missing. 6. Math presentations need to be revised. For instance, where does U come from in L101? Q sometimes takes one input argument but sometimes 2. I might change the score in the end, but I still find the writing not ready for publication, despite its effectiveness. I strongly recommend rephrasing the introduction if the paper gets accepted.
Response to the Reviewer
We thank the reviewer for the response. We are glad that our rebuttal has addressed the reviewer's concerns especially on the research problem. It is also encouraging to hear that the reviewer is open to adjusting the score. The remaining concerns from the reviewer are on the presentation, and we thank the reviewer for clarifying them. Below we address all these points in further detail, and we will revise the paper accordingly. \ *In L21-26, it looks like the authors plan to work on improving the in-distribution accuracy under adversarial learning. But, it turns out to investigate data augmentation with the zero-shot generalization of foundation models (L33).* In this section, our intent was to establish the motivation for DAD. We kindly refer the reviewer to L36, where our motivation is explained as combining the strengths of regularization and foundation models. Before L36, we point out the weaknesses of either approach. In L21-26, we argue that prior regularization techniques like adversarial training reduces in-distribution accuracy (L25) or does not transfer well to out-of-distribution robustness (L26). Therefore, our focus is on distilling foundation models through data augmentation. We realize that the current presentation in paragraph L21-26 could be confusing. We will revise this paragraph to make it clearer by 1) highlighting the deficiencies of current data augmentation methods in particular, and 2) moving the discussion on adversarial robustness to the Related Work section. \ *L36-38, again, is confusing. As mentioned in my previous comment, it gives the impression that the goal is solely to distill large models into smaller ones.* We see the reviewer’s concern. For clarity, it may be helpful to separate the methodology from the goal. Broadly, our goal is indeed to improve out-of-distribution robustness, as the reviewer previously noted. But our novelty and methodology are based on distilling from large models. We will make this clearer by revising L36-38 in the following way: - In this paper, we aim to connect these two lines of work. Our goal is to improve the out-of-distribution robustness of small models without large-scale training. To this end, we introduce a foundation model as a teacher to improve the diversity of data augmentation and directly distill robust representations. \ *Is L40-L46 a finding from the authors or cited from other works? Does this only happen to CLIP?* The finding is from us. The ability to transfer robustness using vanilla knowledge distillation is our finding and unexplored in prior works. The finding does not only happen to CLIP. We observe in Table 8 in the Appendix that this finding generalizes across different teacher architectures. \ *Following (3), Table 7 in the appendix is important evidence to motivate the proposed method, where normal augmented data are unsuitable for distilling CLIP models. It is unclear why the authors leave them in the appendix.* We left this result in the Appendix due to lack of space, but we agree that it is important. So we will add one of the main results of Table 7 (CLIP can distill robustness) as a figure in the introduction, and we will also place it in the experimental section as an ablation for using common data augmentations to demonstrate the need for the generative model.
Response to the Reviewer cont.
**continued from the previous response* \ *The work's main contribution is to leverage discretizers, i.e., VQ-GAN, and the idea is highly built upon the prior work DAT.* We would like to respectfully clarify that although we do draw upon the idea of using VQ-GAN from DAT, we additionally contribute a new setting, knowledge distillation objective for out-of-distribution robustness, and data augmentation generated using teacher gradients. We refer the reviewer to the initial response to concerns about novelty for more detail. *However, relations, detailed comparisons, or motivations are totally missing in this work, making the readers hard to judge the contribution.* - We already provided relations to prior uses of generative models for regularization in L79-83. - The motivation for using a discretizer in our setting can be found in L117-121. - While we do not provide detailed comparisons to other discretizers in the paper, we refer in L145-146 that this is due to the direct use of VQ-GAN from DAT where this is already ablated in Table 5 [1]. In the rebuttal phase, we also previously provided an additional result for Reviewer qyW3 on Stable Diffusion that demonstrates the need for VQ-GAN, and we will add this result as an additional ablation in the revision. We also list the result here: | | ImageNet | V2 | R | Sketch | A | Avg | | --- | --- | --- | --- | --- | --- | --- | | Stable Diffusion | 79.1| 67.8 |45.9 |33.4 | 22.0| 49.6 | | VQGAN | 79.6 | 69.9 | 65.1 | 46.1 | 31.8 | 69.5 | \ *The operation of VQ-VAE is also missing.* We described the operation of VQ-GAN in L145-150. For additional clarity, we will add details in the revised paper about how VQ-GAN is trained, why an image-to-image discretizer is necessary in particular, and how it is used in DAD to generate adversarial examples. \ *Math presentations need to be revised. For instance, where does U come from in L101? Q sometimes takes one input argument but sometimes 2.* We define U as the set of all transformed images (L101-102). We checked the relevant sections and did not find any particular instances where Q takes more than one input, does the reviewer mean $\theta$ ? We use two input arguments for simplicity for the student model $\theta$ in Eqs. 4 and 5. We realize this could be confusing and will expand the equation to include terms for both the clean and perturbed inputs like so. Eq. 4 min $E_{(x,y)\sim P}~[ ~l(\theta(x), y) ~+ $ max $l(\theta(x'), y) ~]$ Eq. 5 min $E_{(x,y)\sim P}~[ ~l(\theta(x), y) ~+ $ max $l(\theta(Q(x')), y) ~]$ \ We hope our response addressed the reviewer’s remaining concerns. We are happy to answer further questions. \ [1] Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi, Shaokai Ye, Xiaodan Li, Rong Zhang, and Hui Xue. Enhance the visual representation via discrete adversarial training. arXiv preprint arXiv:2209.07735, 2022.
Follow up to the Reviewer
Hello, As the discussion period is closing soon, we’d like to follow up and see if the Reviewer has had a chance to consider our response. We hope the Reviewer can raise the score if the concerns were addressed.
Dear authors, Thanks for the detailed follow-up. I have adjusted my score accordingly. And, yes, I meant to $\theta$. After the rebuttal, I still strongly recommend the authors revise the introduction and the paper accordingly, outlining the paper's main focus.
We thank the reviewer for adjusting the score and for providing detailed feedback. We find the comments constructive and will revise the paper accordingly, along with the feedback from other reviewers.
I have carefully read the authors' rebuttal and reviewed the submitted code. I thank the authors for their effort and time in addressing my questions. I have no further comments.
Decision
Accept (poster)