Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains a shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle to identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots"--regions where trojaned classifiers erroneously identify out-of-distribution (OOD) samples as in-distribution (ID). We scan for these blind spots by adversarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data is absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy.
Paper
Similar papers
Peer review
Summary
The paper addresses the problem of detecting trojaned models. The paper proposes a trojaned model scanning method using out-of-distribution (OOD) samples. Specifically, it is observed that trojaned classifiers can erroneously identify adversarially attacked OOD samples as in-distribution (ID) samples. Therefore, the increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. The proposed trojan detection method can be applied in two scenarios: avaliability of clean samples and non-avaliability of clean ID samples. Extensive experiments demonstrate that proposed can achieve state-of-the-art performance compared with other scanning trojan detection methods.
Strengths
The paper proposes a trojan detection method using out-of-distribution samples. This can be applied when no training in-distribution samples is available, which is effective in the real-word applications. The paper conducts sufficient experiments to demonstrate the effectiveness of proposed method. The trojan detection experiments are conducted compared with 7 state-of-the-art scanning methods on 5 datasets across various architectures including CNN and Transformer. The authors present both empirical results and theoretical analysis of proposed method. This is convincing.
Weaknesses
The motivation (especially in Introduction Section) of proposed method is not presented clearly. For example, in line 73-74, the paper introduces the concept of near-OOD samples. However, the difference of far-OOD and near-OOD samples is not clearly explained. Also, the paper writes that "see the visual demonstration in Appendix Section 5". It is obvious there is no Section 5 in Appendix. The paper claims that current methods can not identify models trojaned using adversarial training (e.g. in line 6-7 and line 112-115). However, there is no detailed experiments to show the difference between detecting models trojaned using adversarial training and normal training. Some ablations can be done. For example, the number of used out-of-distribution samples, and the diversity of used OOD samples.
Questions
Could you provide some details of used transformations? Could you compute FID between transformed samples and original samples? The paper only compares with scanning methods. How about other kinds of state-of-the-art trojan detection methods?
Rating
4
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The limitations have been addressed.
Summary
The authors propose a trojan scanning technique that leverages the sensitivity of the network's confidence when near-OOD samples undergo an adversarial attack. The authors argue that the greater variation in confidence can be used to discriminate whether a network has been backdoored, and present extensive experiments to support the effectiveness of their method.
Strengths
- The authors compare with a large number of baselines and a wide variety of datasets and trojan attacks, with standard and adversarial training on 3 architectures. - The concept is intuitive and the method seems to be empirically effective in a wide variety of cases.
Weaknesses
- W0.1 (On writing) The concepts are very simple, but the presentation is hard to follow. Lines 40-89 refer to figures and concepts that will be displayed in following sections. I would suggest to anticipate significantly the figures. Furthermore, I would suggest providing immediately clear definitions of concepts, that are instead introduced in a handwavy way (e.g., blind spots, benign overfitting) before being properly discussed in related works. Probably completely restructuring Sections 1 and 2 to transfer most of the contents of 1 to 2 could help, having a good 'definitions' or 'preliminaries' section in Section 2. - W0.2 (On writing) The formatting of the theorems in the main paper (especially with lack of assumptions) is very unusual and may e confusing. - W1: The scanning procedure may be vulnerable to an adaptive trojan attack. Could the authors show what happens if attackers account for the author's defense in their attack strategy? How much does the effectiveness of their technique go down? The attackers may completely elude the proposed scanning method if they account for it in the design of their attack. - W2: How sensitive are the hyperparameters to the choice of the validation set? How computationally expensive is it to tune them? - W3: Results are adequately reported exclusively for Resnet18, and are not reported exhaustively for other architectures (or their presentation is not really immediately clear).
Questions
- Q1: There must be a typo at line 161, I could not find information about hard transformations and therefore cannot verify claims made about them (Appendix Section 5). The typo "Attention Section" occurs repeatedly throughout the paper. . Q2: When using TinyImageNet, why did the authors not simply filter out the classes overlapping with CIFAR-10? This can be easily done and show the effectiveness of the method without applying G(.).
Rating
6
Confidence
4
Soundness
2
Presentation
1
Contribution
3
Limitations
- L1: The method seems to assume to know the training distribution of the model so that it is possible to find near-OOD samples. However this may not always be possible or easy (near-ness could depend on many factors other than class similarity, e.g. may depend on resolution of the input, color range, other forms of naturally occurring covariate-shifts, types of shortcuts taken by the models etc.). The method significantly improves when this info is accessible. - L2: The previous limitation spills over to the need of having a good validation set. In some cases, the performance can go below the baselines if the validation set is not representative of the training set.
Appreciation for Your Positive Feedback
Thank you for your thoughtful feedback and for taking the time to review our rebuttal! We greatly appreciate your careful evaluation and positive response. Sincerely, The Authors
Summary
This paper proposes a general strategy for distinguishing between trojaned and clean models. The generality of the approach lies in its applicability to various types of trojan attacks, different label-mapping strategies, and its ability to work with and without clean training data. The authors claim that distorted areas of the learned decision boundary in trojaned models, referred to as blind spots, serve as a consistent signature for distinguishing between trojaned and clean classifiers, regardless of the trojan attack methodology. A key characteristic of these blind spots is that samples within these regions are expected to be out-of-distribution (OOD) relative to the clean training data, yet trojaned classifiers mistakenly identify them as in-distribution (ID) samples. To leverage this characteristic, the authors propose using adversarial attacks to perturb a given OOD sample towards an ID direction, followed by computing the difference in ID score (i.e., maximum softmax probability) before and after the adversarial attack. They found that trojaned models exhibit significantly larger differences in these scores. The effectiveness of the proposed detection method was empirically validated against eight trojan attacks and under different levels of access to clean training data.
Strengths
1. The paper is well-written and easy to read. 2. It includes clear explanations, supported by figures, to illustrate the intuition and proposed method. 3. The authors conduct extensive experiments across various types of trojan attacks and different levels of access to clean training data. 4. They propose a simple yet effective method to detect trojaned models. 5. Theoretical analysis is provided, demonstrating that a trojaned neural network is more sensitive to adversarial perturbations.
Weaknesses
As mentioned in the limitation section, the selections of $\epsilon$ and $\tau$ rely on a validation set and a surrogate model. As a result, the quality of these hyperparameters depend on the choices of the validation set and the surrogate model. In particular, when detecting a trojaned model with a new model architecture or trained on a new domain, one might need to tune $\epsilon$ and $\tau$.
Questions
1. Could you clarify what it means for a trojaned classifier to be adversarially trained on poisoned training data? (For example, for poisoning samples generated by a label-consistent trojan attack, what does it mean by adversarially training the trojaned model on these data?) Additionally, are there any empirical results demonstrating the effectiveness of the proposed method in this specific scenario? 2. When a portion of clean training data is not accessible, the authors propose to use Tiny-ImageNet. In addition, it comes to my mind that given the accessibility to the model, one could actually reverse engineer the training data. In that way, you could obtain a portion of fake clean training data, followed by creating near-OOD samples.Consequently, the performance without clean training data could approach that of having a portion of real clean training data, thereby diminishing the performance gap between TRODO-Zero and TRODO. In particular, the reverse engineering of training data has been adopted in many backdoor works, such as [1] (see Figure 3). 3. Headings of paragraphs could be set in a uniform format. Eg., In Line 149, there is a ‘.’ after the paragraph heading; by comparison, in Line 168, there is not. [1] Liu, Y., Ma, S., Aafer, Y., Lee, W. C., Zhai, J., Wang, W., & Zhang, X. (2017). Trojaning attack on neural networks.
Rating
6
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
The performance of the proposed method drops by a large margin (over 10%) when no clean training data is accessible, compared to that with a portion of clean training data.
Summary
The paper proposes TRODO, a method for identifying trojaned classifiers, which relies on the intuition that in presence of a backdoor it should be easier than for clean classifiers to make the model classify an out-of-distribution input as in-distribution by adding an adversarial perturbation. In practice, a PGD attack is run on OOD images to increase the confidence of the classifier predictions, and the average increase is used to detect backdoored models. In the experimental evaluation, the proposed method outperforms the baselines, being effective against several backdoor attacks even when combined with adversarial training.
Strengths
- The proposed method is very general, as it doesn't make assumptions on the type of backdoor attacks or target architectures. Moreover, the paper proposes variants with and without access to training data. Finally, the proposed approach appears less expensive than the baselines (Table 2). - The experimental results support the proposed method, which is effective against several backdoor attacks even when combined with adversarial training.
Weaknesses
- The proposed method seems to have brittle elements: - the PGD-10 attack used for computing the signature might be considered weak (only 10 iterations), then not able to fully optimize the target loss. Then, a stronger attack could further increase the confidence of the clean classifiers on OOD points, making them more similar to trojaned one (since the confidence is upper bounded, one could get the same score for all classifiers in the worst-case). Thus, the effectiveness of TRODO seems to rely to some extent on the attack not fully optimizing the target loss (which might be even exploited to bypass the detection mechanism). - the confidence of any classifier might be adjusted post-training by e.g. temperature rescaling without changing its decisions. In this way, it seems possible to bypass the detection by making a model under-confident. - adversarial training variants, e.g. on OOD data [60] or to have uniform predictions far from ID data [A], have been explored in prior works, and might be used to counter the proposed scanning scheme. - It's not clear how the effectiveness of TRODO correlates with the strength of the backdoor attacks. For example, one can imagine that using a lower poisoning rate might make the attack less detectable (but less effective). [A] https://arxiv.org/abs/1910.06259
Questions
- Why using a left-truncated normal distribution for estimating $\tau$ when the score S can take values only on a specific range (confidence is upper bounded by 1)? - While the proposed method provides good experimental results, I think it's important to address the concerns about its robustness (see above).
Rating
3
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations are partially addressed.
Response to Reviewer PnTa
> **Am I missing something?** * We believe there has been some misunderstanding. When we mentioned that _'the classifier remains more confident in in-distribution samples compared to OOD samples,'_ we were referring to **Fact-A**. We would like to clarify that the principle behind our signature remains valid even when temperature rescaling is applied. The logical flow of our principle is as follows: **Fact-A ⇒ Fact-B ⇒ Fact-C ⇒ Fact-D ⇒ Fact-E ⇒ Fact-F ⇒ Fact-G**. * Furthermore, as indicated in Fact-A, the softmax output tends to resemble a uniform distribution for OOD samples. As a result, applying temperature rescaling will affect all logits equally and **will not** alter this uniform shape. Consequently the ID score for clean OOD samples would still be very low (e.g., approximately $\frac{1}{N}$, where $N$ is the number of ID classes). However, by perturbing these samples, the ID score can increase (up to one). Therefore, our signature would not be arbitrarily small as mentioned; it remains within the range of ([0,$1-\frac{1}{N}$]). **Fact-A**: A classifier remains more confident in IDs compared to OODs. Specifically, the output softmax of a classifier for OODs tends to be more uniform, while for IDs, it is more concentrated. (confidence: maximum softmax probablity) [1]. **Fact-B**: Previous research has primarily focused on perturbing IDs, shifting them, and using this as a signature to distinguish between clean and trojaned classifiers [2,3,4,5,6,7,8,9,10]. **Fact-C**: In scenarios where classifiers have been adversarially trained, perturbing ID samples for signature extraction becomes less effective. This is because adversarial training enhances the classifier's robustness to such perturbations, reducing their impact [9,10]. **Fact-D**: Instead of perturbing IDs, perturbing OODs toward the in-distribution is a viable approach, as classifiers are vulnerable to this shift. This approach remains effective even for adversarially trained classifiers, as they are still susceptible to perturbed OODs. This finding is also supported by parallel research in OOD detection [11,12,13,14]. **Fact-E**: Trojaned classifiers learn decision boundaries that include 'blind spots,' which are intuitively regions along the in-distribution boundary that have been altered to overfit on poisoned training data (data with triggers), thereby changing the geometry of the boundary. **Fact-F**: Perturbing OODs toward the classifier's decision boundary increases their ID scores, and this effect is particularly significant in trojaned classifiers. This is because the perturbations can exploit the blind spots within the decision boundary, mimicking the triggers used during the trojan attack. **Fact-G***: We use the difference in ID scores between a benign OOD sample (without perturbation) and an adversarially perturbed OOD sample as a signature to differentiate between clean and trojaned classifiers. --- >**What was the OOD...?** Following their reported setting, we used Tiny ImageNet as the out-of-distribution dataset for training. --- >**In general, these methods.. .** Regardless of the adaptation strategy, existing classifiers are vulnerable to adversarial attacks on OODs, particularly when these samples are close to the in-distribution boundary. This vulnerability, as demonstrated recently by (Lorenz Deciphering, 2024), (Fort Adversarial Vulnerability, 2022), and others [11,12,13,14], limits them to act as counters to our method. To further illustrate this, we conducted additional experiments using RATIO as an OOD detector, where the MSP is employed as the ID score. In the first scenario, CIFAR-10 serves as the ID dataset and CIFAR-100 as the OOD, and vice versa in the second. While the method achieves good performance in clean scenarios , in adversarial scenarios—where perturbations are added to shift OODs to in-distribution and vice versa—its performance drops to below random detection, highlighting its vulnerability to perturbed OODs. | Method\Benchmark | _CIFAR-10 vs.CIFAR-100_ |_CIFAR-100 vs. CIFAR-10_| |-|-|-| | | AUC/AUC* |AUC/AUC* | | RATIO | 81.3/14.8 |68.5/9.0 | --- >**I think the additional ...** In the adaptive attack scenario, we evaluated various label mappings and adaptive strategies. The worst performance decrease we observed was 12% (from 86.0% to 74.1%). Despite this, our method still outperforms previous detection methods that were not subjected to adaptive attacks. For instance, UMD achieves 57.7%, while our approach achieves 74.1%. [1] Hendrycks, A baseline, 2016 [2] Xiang, UMD: Unsupervised , 2023 [3] Wang, Mm-bd: , 2024 [4] Wang, Neural, 2019 [5] Liu, Abs:Scanning, 2019 [6] Shen, Backdoor scanning for, 2021 [7] Guo, Tabor, 2019 [8] Hu, Trigger hunting, 2022 [9] Edraki, Odyssey, 2021 [10]Zhang, Cassandra, 2021 [11] Chen Robust OOD 2020 [12] Azizmalayeri, Your Detector 2022 [13] Chen, Atom ,2021 [14] Mirzaei, RODEO, 2024
> Furthermore, as indicated in Fact-A, the softmax output tends to resemble a uniform distribution for OOD samples. As a result, applying temperature rescaling will affect all logits equally and will not alter this uniform shape. Unless the softmax output is exactly uniform (i.e. all logits are identical), which seems very unlikely, temperature will change the distribution. > However, by perturbing these samples, the ID score can increase (up to one). Therefore, our signature would not be arbitrarily small as mentioned; it remains within the range of ($[0, 1 - 1/N]$). I meant arbitrarily small within its range of course (as the ID-Score after the attack cannot be lower than on the clean input). The point of using temperature >> 1, i.e. making the model under-confident, is that even when the attack is applied it will not increase much the confidence, in particularly since the hyperparameters of the attack such as $\epsilon$ and number of steps is fixed and calibrated on standard models. Also, one can consider the other extreme case with temperature = 0 (in practice, values close to 0 are sufficient), i.e. the softmax becomes the argmax function. If the argmax of the logits is unique, then the softmax output is a one-hot vector, and even for OOD points the difference of ID-Scores will be 0. This seems to bypass the detection mechanism. Is this correct?
Response to Reviewer PnTa
We thank the reviewer for their thoughtful review and valuable feedback. * A key requirement for deploying deep neural networks in real-world applications is calibrated or approximately calibrated behavior on input samples. A miscalibrated classifier, particularly in cases where $T→0$, $T>>1$, can lead to poor decision-making. For example, in medical diagnosis, overestimating disease probability can result in unnecessary treatments, while underestimating it may lead to missed diagnoses. Miscalibrated models are especially unreliable in high-stakes scenarios such as finance, healthcare, or autonomous systems, where accurate probability estimates are crucial. Furthermore, the backdoor attack/defense literature generally assumes that Trojaned classifiers behave normally on samples without triggers, similarly to standard classifiers. Since standard classifiers typically exhibit approximately calibrated outputs, we implicitly assume that the Trojaned classifier would also retain this characteristic. * We acknowledge the challenges posed by the extreme temperature scenarios, specifically when $T→0$ or $T>>1$, as our ID-score would converge to zero in both cases. However, we believe this issue can be addressed with minimal modifications. In the backdoor attack setup (as detailed in our threat model), it is commonly assumed that the attacker has access to the input model. To mitigate the issue, we propose a minor extension to TRODO: applying its own softmax function to the logits of the input classifier instead of relying on the input classifier's final softmax output. This approach could help counter the adversarial temperature settings highlighted by the reviewer. * Moreover, since TRODO and TRODO-Zero both utilize an OOD set, another minor extension could involve evaluating the classifier's softmax output on these samples. If the output significantly deviates from a uniform distribution (e.g. measured by Kullback–Leibler divergence), the model could be rejected as extremely miscalibrated due to the attacker's manipulation. To further address your concern, we evaluated TRODO's performance on an all-to-one label mapping task using the CIFAR-10 and GTSRB datasets across different temperature values, while keeping other components unchanged. The results show that our designed detector effectively handles a reasonable range of temperature values, maintaining consistent performance throughout. | CIFAR10 | $T=0.5$ | $T=0.7$ | $T=1$(default) | $T=1.2 $ |$T=1.5$ | $T=2$ | |-|-|-|-|-|-|-| | ACC/ACC* | 90.5/87.1 | 91.2/87.3 | 91.0/88.4 | 89.4/86.2 | 88.4/85.7 | 87.1/85.3 | | GTSRB | $T=0.5$ | $T=0.7$ | $T=1$(default) | $T=1.2 $ |$T=1.5$ | $T=2$ | |-|-|-|-|-|-|-| | ACC/ACC* | 95.2/92.7 | 94.8/93.0 |96.6/93.2 | 95.7/92.1 |94.3/92.8 |92.7/91.5 |
Thanks for the additional reply. > A miscalibrated classifier [...] can lead to poor decision-making... In classification tasks as those reported in the paper, calibration is not a factor, since confidence is not used for classification (only argmax). > Since standard classifiers typically exhibit approximately calibrated outputs... I think this is not in general precise, that is it's not clear whether classifiers are calibrated before applying post-processing calibration techniques (see e.g. https://arxiv.org/abs/1706.04599). > However, we believe this issue can be addressed with minimal modifications... This might be true, but should be experimentally tested and discussed in the paper. I think this simple approach (modifying the confidence of the model via temperature rescaling), which doesn't require designing new backdoor attacks, points to weaknesses of the proposed detection method, which can potentially be exploited by more sophisticated adaptive attacks. I think this can't be simply dismissed assuming models are well calibrated, and should be discussed in the paper. Overall, the rebuttal has confirmed that the proposed method, in the current form, is susceptible to temperature rescaling and (at least partially) other adaptive attacks ($L_{adaptive1}$). Thus, I think the paper would require significant improvements, at least discussing the current limitations and potential countermeasures. Then, since the original main concerns remain, I will vote for rejecting the paper.
Response to Reviewer PnTa
We thank the reviewer for their feedback. We believe the discussion around temperature scaling has diverged from our primary focus. Our intention was to highlight the extreme cases of a miscalibrated classifier, such as those producing one-hot or uniform outputs, which, as the reviewer suggested, are uncommon in real-world scenarios due to their lack of explainability. This is why we did not address them in our paper. While these scenarios may be relevant in theoretical contexts, they are not directly applicable to practical situations. Nonetheless, our adaptive attacks demonstrate that even in the worst-case scenarios, TRODO performs consistently. To **fully address** the reviewer's concerns, as they **agreed**, we will consider applying the softmax function ourselves rather than relying on the classifier's softmax, and we will discuss this in our paper. >**This might be true, but should be experimentally tested and discussed in the paper.** Our experiments were conducted under the assumption that temperature scaling was not applied ($T=1$). Therefore, applying softmax ourselves instead of using the classifier’s built-in softmax would not change the results, and the experiments would yield identical outcomes. >**Overall, the rebuttal has confirmed that the proposed method is susceptible to.... adaptive attacks** We believe that a fair comparison should also consider the performance of other methods under adaptive attacks for a evaluation. However, even under strong adaptive attacks, our method continues to demonstrate superior performance compared to existing scanning methods. We remain open to further discussion to address any additional concerns the reviewer may have. Sincerely, The Authors
Summary
This paper points out a limitation of existing backdoor model scanning methods: They fail to detect backdoored models trained with adversarial training. It propose a new backdoor model scanning method by utilizing adversarial shifts in Out-of-distribution samples. Experiments on MNIST, CIFAR-10, GTSRB, CIFAR-100, and Pubfig show the effectiveness of the proposed method.
Strengths
* The investigated problem is interesting. * The motivation of this paper is good. * This paper provides theoretical analysis to support the proposed observation and method. * The proposed scanning method is general to different types of backdoor attacks.
Weaknesses
* The paper states that experiments were conducted using ResNet18, PreActResNet18, and ViT-B/16 models. However, Table 1 only presents results for ResNet18. While the paper claims that results for other models are included in Appendix Section M, but this section also only contains ResNet18 results. It would be beneficial to explicitly present the detection accuracy for each model across various attack scenarios. * The discussion about the adaptive attacks to the proposed method is missing, where the attacker knows about the proposed defense strategy and actively tries to evade or overcome it. For example, the adaptive attacker might able to add a loss during the backdoored model construction phase to reduce the change of the ID-Score on the backdoored models. * While theoretical analysis of the proposed method is highly recommended, the paper would benefit from more intuitive explanations of this analysis. There appears to be a lack of detailed clarification regarding the connection between the theoretical analysis and the proposed method. For instance, the relationship between Theorem 2 and the fundamental principles of the proposed method is not clearly stated. Providing more intuitive explanations of these theoretical analysis would strengthen the paper.
Questions
See Weaknesses.
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The limitations have been discussed in this paper.
Summary
The paper introduces a novel trojan scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots," where trojaned classifiers mistakenly identify out-of-distribution (OOD) samples as in-distribution (ID). The method scans for these blind spots by adversarially shifting OOD samples towards in-distribution, using the increased likelihood of these perturbed samples being classified as ID as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers, and applicable even when training data is absent.
Strengths
The writing is clear and the pictures are crisp and clear.
Weaknesses
Undefined Threat Model: The threat model is not clearly defined, and the adversary's capabilities are not explicitly listed. Lack of Comparison with SOTA Baselines: There is an absence of comparison with SOTA baselines, such as FreeEagle. The results listed in Table 2 (baseline-NC) differ significantly from those in FreeEagle's Table 4. Additionally, many baselines from FreeEagle's Table 4, such as STRIP[2] and ANP[3], are not discussed. [1]Fu, C., Zhang, X., Ji, S., Wang, T., Lin, P., Feng, Y., & Yin, J. (2023). {FreeEagle}: Detecting Complex Neural Trojans in {Data-Free} Cases. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 6399-6416). [2]Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, S. (2019, December). Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference (pp. 113-125). [3]Wu, D., & Wang, Y. (2021). Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 16913-16925.
Questions
1. Can you clarify your threat model? 2. Why do you have so many missing baselines?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Design is too simple
Thank You for Your Positive Feedback
Thank you for your positive feedback and for considering a higher score for our work! We will ensure that these experiments are incorporated to further enhance the manuscript. Sincerely, The Authors
Response to rebuttal
I have read the authors' rebuttals and the other reviewers' opinions. I think several of the concernes raised by other reviewers are fair, but the authors were convincing in their rebuttals. I appreciate the clarification, additional information and experiments performed by the reviewers, especially the introduction of an analysis of their method under adaptive attacks. While this may not be a final guarantee about the robustness of the proposed technique and the limitations pointed out are quite strong, I think this is reasonable and what an attacker would try out first, and the relative robustness of the technique to it indicates a significant research effort would be required in order to further degrade the performance of this technique. Therefore, I am happy to increase my score.
W1: Thanks for the clarification. Q1: I still have some confusions about adversarially trained models. "To clarify, the training dataset is initially compromised using a backdoor attack. The classifier is then adversarially trained using the PGD (Projected Gradient Descent) attack method."—Could you clarify: 1) the data used for this adversarial training, 2) along with the objective function, and 3) the motivation for this adversarial training? Thanks. Q2: 1) So according to your claim, can we say that the samples generated by my mentioned's method are less effective than some randomly selected samples from a OOD dataset, e.g. Tiny ImageNet? 2) Is FastDFKD (Fang Data-free 2022) method using models to recover some training data? 2) I think my concern for using models to recover some training data when there is no clean training data arises from the large accuracy gap between TRODO and TRODO-Zero. (My intuition is that the samples generated in this way are more effective than some randomly selected samples from a OOD dataset) Thus, the comparison should be with TRODO-Zero, instead of TRODO.
Thanks for your feedback. Here are the answers to your questions: Q1: 1) Each column in the table 1, above, shows the original training data. The data is first altered by the backdoor attack by adding the backdoor to the input and changing the corresponding output labels based on all-to-one (upper half) or all-to-all (lower half) . Then, adversarial training is applied on the entire dataset including the poisoned and clean samples. Here, each training data, which might be poisoned, undergoes an adversarial attack during the training. As in the regular adversarial training, we only perturb the input and keep the ground truth label unchanged. 2) The objective function in the adversarial training is the regular cross-entropy (line 593). 3) The main motivation for evaluating scanning methods on such models is that many previously proposed signatures for trojan scanning experience a detection accuracy drop in the case of an adversarially trained model (lines 37-39). For instance, in the table 1 above, UMD's detection acc. drops by almost 20% points on MNIST and CIFAR-10 on the adversarially trained models compared to the standard models (compare ACC with ACC*). The possible reason behind this effect is that adversarial training makes shortcut learning difficult, potentially leading to harder trigger reverse-engineering and identification. While previous scanning methods struggle with this issue, our proposed method does not rely on such practices and shows insignificant drop under adversarially trained models. Q2: 2, 3) Please note that FastDFKD (Up to 100× Faster Data-free Knowledge Distillation by Fang et al.) is an efficient method of generating synthetic samples given access to a trained model (see Fig. 3 of that paper). Here, we use this model to create surrogates of the training samples and use them in our method TRODO-Zero to enhance it. So indeed the row mentioning "FastDFKD" *is* the TRODO-Zero with the synthesized samples using FastDFKD. As can be seen, there is still a large gap between the two TRODO and TRODO-Zero (it even got slightly worse). The first row, TrojaNN, (Trojan Signatures in DNN Weights, by Fields et al.), which also assumes no access to the training data is included as a baseline. 1) Like we mentioned, we hypothesize that the generated samples, through methods such as FastDFKD, could contain shortcuts, potentially imperceptible, making them *unideal* OOD instances, compared to the clean tiny-ImageNet samples that lack such artifacts. Please note that such artifacts would help the classifier to classify the intended synthetic OOD sample as ID more *easily,* reducing the effectiveness of the method, because we take the gap between the ID score of the OOD sample before and after the attack as our detection score (line 179). Therefore, while your idea could generally be beneficial in other methods, it would be less so in our specific method that relies on measuring ID-ness of synthetic OOD samples before and after the attack.
Q1: Thanks for the detailed clarification. Now I understand the motivation for the adversarial training. I also appreciate the good detection performance of your method compared with others under the adversarial training scenario. Q2: Thanks for your clarification, especially "the row mentioning "FastDFKD" is the TRODO-Zero with the synthesized samples using FastDFKD.", which solves my confusion. I also appreciate your intuitive explanation in analyzing the bad performance of using the generated training data. I will keep my score. Thanks!
Thanks for the detailed response and additional experiments. > **W1.2:** "the classifier remains more confident in in-distribution samples compared to OOD samples" If I understand it correctly, only OOD samples are used to compute the signature. Then, I think the absolute difference $\textrm{ID-Score}_f (x_i^{OOD*}) − \textrm{ID-Score}_f (x_i^{OOD})$ could be (most likely) made arbitrarily small with temperature rescaling, so that it's in the range of the signature of a clean classifier. Similarly, since the confidence is upper bound by 1, also an over-confident classifier which has near 1 confidence on OOD data would lead to very small differences between adversarially perturbed and clean OOD points (since the ID-Score cannot increase much). Am I missing something? > **W1.3:** "Improving robustness on adversarially out-of-distribution (OOD) samples was not their main purpose" RATIO loss includes a term which is a robust loss on OOD data, then it directly optimizes adversarial robustness on OOD points. What was the OOD data used for training the RATIO models? In general, these methods show that it is possible to control the (adversarial) confidence on OOD data (but might require some adaptation to the specific task of bypassing the detection mechanism). > common response I think the additional experiments with the new losses show that adaptive attacks have the potential to bypass the detection mechanism of TRODO: in fact, $L_{adaptive1}$ improves 12% compared to $L_{default}$ in the All-to-All setup, which seems significant.
Thank you for the clarification and additional experiments. The author should consider adding this content to the article. I have decided to raise my score.
Appreciation for Your Feedback and Review
Thank you for your valuable review and positive feedback! We are pleased to hear that your concerns have been addressed. Warm regards, The Authors
Dear Reviewer d7KJ, As we approach the discussion period deadline, we kindly request that you review our rebuttal, in which we have aimed to thoroughly address your comments. If you have any further concerns, we would be most grateful if you could bring them to our attention, and we would be pleased to discuss them. Sincerely, The Authors
Decision
Accept (poster)