Indirect Gradient Matching for Adversarial Robust Distillation

Reply to Reviewer Pxd3.

Thank you for your detailed review and constructive feedback. We appreciate your thorough assessment of IGDM and the opportunity to clarify certain points. **W1. The paper does not make it clear how this way gradient matching is "indirect." it is not clear why this is "indirect" gradient distillation. There is no comparative picture presented to clarify the comparison/contrast between Indirect Gradient and Direct Gradient distillation.** **A.** We apologize for any ambiguity in our terminology. Our primary goal is to perform adversarial distillation from a highly robust teacher model, specifically by distilling information from the input gradients, which exhibit critical properties in adversarial training. We use the term “indirect” because our loss design does not calculate the gradient directly; instead, we align gradients through a loss that approximates this effect without explicitly computing the gradient. In Appendix A.4, we provide a comprehensive comparison with methods that perform direct gradient calculation, which further clarifies this distinction. **W2. It is not explicitly mentioned or clear that they run the other (SOTA) algorithms themselves or that the results were taken from the reported results. If experiments were re-run, other hyperparameter setting descriptions were not given or not referred to which hyperparameter setting were used, e.g., home many epoch os training.** **A.** For all experiments except for Table 7, we ran the baseline algorithms ourselves, strictly following their best hyperparameter settings and training epochs for a fair comparison. We referenced both their official code and the best-performing parameters specified in their papers. Detailed descriptions of experiment settings, including training epochs, are provided in Appendix B.2. For example, for parameters like $\lambda$ in RSLAD and AdaAD, we used $\lambda = 1$ as recommended in their original settings. **W3. There is no indication of making code available for reproducibility is indicated in the paper.** **A.** We have already attached our code in the supplementary material. We apologize for not indicating this in the paper. **Q1. Figure 2a says that the "remainder proportion" of the adversarial robust teachers is shown in Table 1** **A.** The caption of Figure 2a is wrong. Sorry for our mistake. We have updated in revision pdf (see blue texts), and the remainder proportion is written in the line number 183-185 of our paper (0.012,0.012,0.016) **Q2. It has not been demonstrated how preparation values 0.012 and 0.016 have arrived.** **A.** the proportion values 0.012, 0.012, 0.016 come from equation (2). As you pointed out, Table 1 is about the accuracy of teacher model. sorry again for our mistake, we have updated our paper. The proportion is calculated as ratio of remainder and total value, i.e., $\frac{\|\|f(\textbf{x} + \boldsymbol{\epsilon}) - f(\textbf{x}) - \left( \frac{\partial f(\textbf{x})}{\partial \textbf{x}} \right)^T \boldsymbol{\epsilon} \|\|}{\|\|f(\textbf{x} + \boldsymbol{\epsilon})\|\|}$. **Q3. It is unclear how this method differs from other methods that do gradient distillation.** **A.** We provide a detailed comparison of other methods that perform gradient distillation within the adversarial distillation context in Appendix A.4.

Reply to Reviewer yGhV.

Thank you for your detailed review and for highlighting both the strengths and areas for improvement in our work. We appreciate your constructive feedback and the opportunity to clarify certain aspects of our paper. **W1. Comparative analysis with existing first-order taylor expansion-based methods.** **A.** Thank you for raising this point. Existing methods in adversarial training [d-f] often aim to achieve low curvature through explicit regularization techniques. In contrast, our approach is distinguished by the **gradient matching** strategy, which is designed to effectively imitate a highly robust teacher model. Instead of directly enforcing low curvature, we leverage the inherent low-curvature property of the teacher model when aligning gradients, using only the output differences. Our focus lies in achieving robustness within the adversarial distillation framework, rather than pursuing low curvature explicitly as in existing adversarial training methods. While incorporating additional losses that emphasize low curvature might further enhance performance—and could even be beneficial across other baseline methods—our loss design intentionally shifts focus to a unique perspective that differentiates it from existing techniques. We hope this clarifies the novelty and intent of our approach compared to methods based on first-order Taylor function expansions. **W2. Clarification on internal loss for generating adversarial perturbations** **A.** We appreciate your suggestion to provide additional details on the internal loss used to generate adversarial perturbations. As in Appendix B.1, we employed the same inner loss functions as outlined in each original paper: PGD attack on the student model for PGD-AT, ARD, and IAD; TRADES attack on the student model for TRADES; RSLAD inner loss for RSLAD; and AdaAD inner loss for AdaAD. Furthermore, in the methods incorporating IGDM, we followed the inner maximization method specified in the original adversarial training or adversarial distillation method. For formulaic representation as follows, PGD attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{CE} (f_S (\mathbf{x}+\boldsymbol{\delta'}), y)$ TRADES attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S (\mathbf{x}+\boldsymbol{\delta'}), f_S (\mathbf{x}))$ RSLAD attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S (\mathbf{x}+\boldsymbol{\delta'}), f_T (\mathbf{x}))$ AdaAD attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S (\mathbf{x}+\boldsymbol{\delta'}), f_T (\mathbf{x} +\boldsymbol{\delta'}))$ Here, $\text{CE}$ represents cross-entropy loss, $\text{KL}$ represents KL-divergence loss. **Q1. What would be the effect if the adversarial perturbations are generated using Equation 9?** **A.** We also experimented with using Equation 9 to generate adversarial examples; however, this did not lead to significant performance improvements. This outcome may be due to the necessity of additional hyperparameter tuning and adjustments to effectively utilize Equation 9 for attacks. Designing a more effective inner maximization process remains an avenue for future work. **W3. Missing related works** **A.** Thank you for pointing out recent related works on adversarial distillation. We apologize for the oversight and will update the paper to include citations. Here is our perspective on each paper: - *Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap (ECCV 2024)* : This is a concurrent paper that was published shortly before we completed our manuscript, which is why it was not included initially. Currently, there is no official code available, so we are unable to reproduce their results for direct comparison in our main paper. - *PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor (CVPR 2024)* : This approach differs slightly from traditional adversarial distillation in that it uses a peer model instead of a teacher model, training both the peer model and the student together. Due to this setup, PeerAiD does not utilize a strong, robust teacher, which results in lower reported performance metrics. Because of these fundamental differences in the experimental setup, we decided not to include PeerAiD in our main table for direct comparison. - *Improving Adversarial Robustness via Information Bottleneck Distillation (Neurips 2023)* : First, this paper has not been widely cited by other adversarial distillation studies. We believe this may be due to limited code availability, which affects reproducibility. Additionally, its absolute performance are lower than those of more recent work, such as AdaAD, so we decided not to use it as a baseline.

Reply to Reviewer avUv.

Thank you for your thoughtful review and for raising insightful questions about our work. We appreciate your recognition of IGDM’s novel approach and its effectiveness in improving adversarial robustness through extensive experiments. **Strength-Q1. Addressing concerns about theoretical assumptions, does this method really simulate the gradient of the network?** **A.** We acknowledge your concern regarding the assumption of local linearity in our gradient distillation approach. While previous studies [1] attribute the fragility of neural networks to their linear properties, we want to clarify that [1] refers to linear networks, which focus on the *network weights*. Our observation, however, is based on the linear property *with respect to the inputs*, not the network weights. The linear property with respect to inputs is observed as shown in Figure 2-(b). To address your question, we also numerically verify that our method simulates the gradients of the network, as shown in Table 4. Specifically, we calculate the gradients with respect to the input for both the teacher and student networks. We then observe that the gradient distance and gradient cosine similarity become closer, demonstrating that our method effectively simulates the gradients with respect to the inputs of the network. **W1. If other methods also use the same teacher model, can they also achieve good robustness?** **A.** We respectfully disagree with the assertion that our comparisons are unfair. To clarify, in all of our experiments, we used the **same teacher model** consistently across different methods within each experimental setting. More detailed reasons are as follows: First, as shown in Table 1, for CIFAR-10 and CIFAR-100, we used publicly available models from RobustBench, ensuring accessibility to all researchers. For SVHN and Tiny-ImageNet, since prior works like RSLAD and AdaAD do not report experiments, we trained teacher models ourselves and plan to make these models publicly available. Additionally, the CIFAR-100 LTD model in Table 1 is the exact same teacher model used in AdaAD, ensuring consistency across methods. Thus, our choice of teacher models was not intended to artificially boost IGDM’s performance by cherry-picking but rather to ensure a fair comparison based on widely recognized or directly comparable models. Second, as shown in Figure 4, we used teachers which are publicly available models from RobustBench. Note that all the baselines utilize the **same** teacher to ensure fair comparison. While a stronger teacher generally improves student performance, we observe cases—such as in Table 3 vs. Table 13 on CIFAR-10—where a lower-performing teacher yields higher distillation outcomes. This suggests that teacher robustness does not always directly correlate with student performance. We find this observation particularly interesting, as it opens up new avenues for future work to explore how teacher model robustness impacts student performance in adversarial distillation. **W2. Placement of gradient discussion in the main text.** **A.** We appreciate the suggestion to bring the discussion of direct gradients matching into the main body of the paper. We agree that this point is indeed important for the motivation of the paper and would ideally be included in the main context. However, as it is tangential to the specific details of the IGDM module itself, and due to page limitations, we place it in the Appendix. **Q1. Inconsistency of manifold spaces between teacher and student.** **A.** As you pointed out, the manifolds of the teacher and student models are different, which means their gradient directions may not perfectly align, and the adversarial samples they generate will not be identical. We did not claim that the adversarial samples of the teacher and student models are the same. Rather, we use adversarial samples generated by the student model to match its gradients with those of the teacher model. This gradient matching enables the student model to better imitate the teacher, as shown in Equation (10). By aligning the gradients, we achieve point-wise alignment between the student and teacher, as demonstrated in Table 5. Figure 1(a) provides a conceptual illustration of this point-wise alignment, which is also numerically verified in Table 5. **Q2. Why does the remainder proportion of the model not increase in the early stages of training?** In Figure 2, since we plotted the remainder proportion per epoch, it appears lower in the early stages of training. If we were to plot it per step, the remainder proportion would be higher in the very early stage. Our experimental findings indicate that adversarial training results in a lower remainder proportion compared to natural training, even at the beginning of training. We believe this lower remainder proportion is a characteristic of adversarial training, which we have observed experimentally.

reply to the response

Thanks for the author's response, which solved some of my concerns. I still have a few questions. 1. For the teacher model, I checked the results of the original RSLAD paper, and its teacher model (WideResNet-34-10) has an accuracy of 26% under AA attack. The student model can achieve an accuracy of 26.7%, while in this manuscript, a more powerful teacher model (~38%) is used. But the defense of the student model is not improved; it is still 26.7 in Table 2. 2. As Reviewer Pxd3 mentioned, I don't think the author's method fits the gradient. I think the proposed method still fits the model's output distribution in the form of residuals.

Reply to Reviewer avUv.

Thank you for your thoughtful questions and for bringing up this important point. **Q1-1. I checked the results of the original RSLAD paper, and its teacher model (WideResNet-34-10) has an accuracy of 26% under AA attack.** **A.** We would like to clarify a point regarding the teacher model used in the original RSLAD paper. On page 5 of the RSLAD paper, it explicitly states: * "For CIFAR-100, we use the WideResNet-70-16 model provided by Gowal et al." in the section 4.1. Additionally, we verified that the official code for RSLAD also utilizes the WideResNet-70-16 teacher model for CIFAR-100. (Reference: [RSLAD GitHub Repository](https://github.com/zibojia/RSLAD/blob/c40e04f6c4edcd87431d0250b168d503b1293745/resnet18_rslad_cifar100.py#L43C1-L43C70)). The Auto Attack (AA) performance of the WideResNet-70-16 teacher model on CIFAR-100, as reported in the original paper and verified by us, is **30.03%**, not **26%**. We hope this clarifies any confusion regarding the teacher model and its performance in the original RSLAD experiments. **Q1-2. the defense of the student model has not improved; it is still 26.7** **A.** Regarding the teacher model, we refer to the RSLAD paper (Chapter 4.4, "How to Choose a Good Teacher?"), which states: * ”We find that the student’s robustness does not increase monotonically with that of the teacher’s; instead, it first rises then drops. We call this phenomenon robust saturation.” This phenomenon is clearly illustrated in Figure 4 of the RSLAD paper on CIFAR-10, where the WideResNet-34-10 teacher model with an AutoAttack robustness of 53.08% achieves the highest student robustness (\~52%). Interestingly, using a stronger teacher model, such as WideResNet-76-10 with an AutoAttack robustness of 57.20%, results in lower student robustness (\~51%). Therefore, the argument that our RSLAD results should improve from 26.7% simply because we used a stronger teacher does not hold, as the original RSLAD findings already establish this inconsistency. The phenomenon of **robust saturation** highlights that a more robust teacher does not necessarily lead to a more robust student, as discussed in the original RSLAD paper. In contrast, our method clearly demonstrates performance improvement over RSLAD under the same teacher setting. This is a key distinction: while RSLAD fails to fully leverage the increased robustness of a stronger teacher, our approach successfully overcomes this limitation. For instance, using the BDM-AT teacher (AA: 38.83%), RSLAD achieves an AA of 26.76%, whereas RSLAD+IGDM achieves a significantly improved AA of 29.78%. **Q2. I don't think the author's method fits the gradient.** **A.** In Table 4, we provided quantitative evidence demonstrating that incorporating our module enhances the gradient matching with the teacher. Additionally, the results in Tables 17 and 18 further validate this point, which shows that as the weight of IGDM increases, the degree of gradient matching (GD, GC) also improves. Furthermore, Equation (4) offers a mathematical explanation, showing that the residual effectively contains gradient information. We supported this with both numerical data in Table 4 and theoretical analysis, including clarification through Taylor expansion that the residual captures gradient information, even if approximated through the output. We would appreciate any further insights from the reviewer if there are still concerns about the validity of these results, despite the clear evidence presented. Reviewer pxd3 has kindly acknowledged that all concerns have been addressed. If there are additional points for discussion, we would be grateful to hear them. Thank you for your thoughtful consideration.

Reply to the response

Thanks for the author's response. Thanks to the author for pointing out the information about the teacher model, and I apologize for that. Actually, I am confused as to why the student model did not improve with a more robust teacher model, even though RLSAD proposed robust saturation for this phenomenon, but there is no similar phenomenon in other distillation methods. In the submitted manuscript, you just used a 28x10 WideResNet, IDGM got excellent results, but RLSAD did not improve.

Reply to Reviewer ECU4.

Thank you for your positive evaluation and thoughtful feedback on our work. We appreciate your recognition of IGDM’s innovative approach in aligning input gradients between teacher and student models, as well as your comments on the clarity and accessibility of the paper. Your insights will be invaluable as we continue to improve and broaden the scope of IGDM. **W1. Validation on more complex models and tasks could provide more universal insights for future research** **A.** We acknowledge your suggestion to evaluate IGDM on more complex models, such as ViT or CLIP, and on more advanced tasks like detection and segmentation. We agree that validation on more diverse architectures and complex tasks would further strengthen IGDM’s general applicability. Expanding our experiments in these directions is a priority for future work, as it will provide additional insights into IGDM’s robustness and scalability.