Library

Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack

Deep neural networks face persistent challenges in defending against backdoor attacks, leading to an ongoing battle between attacks and defenses. While existing backdoor defense strategies have shown promising performance on reducing attack success rates, can we confidently claim that the backdoor threat has truly been eliminated from the model? To address it, we re-investigate the characteristics of the backdoored models after defense (denoted as defense models). Surprisingly, we find that the original backdoors still exist in defense models derived from existing post-training defense strategies, and the backdoor existence is measured by a novel metric called backdoor existence coefficient. It implies that the backdoors just lie dormant rather than being eliminated. To further verify this finding, we empirically show that these dormant backdoors can be easily re-activated during inference, by manipulating the original trigger with well-designed tiny perturbation using universal adversarial attack. More practically, we extend our backdoor reactivation to black-box scenario, where the defense model can only be queried by the adversary during inference, and develop two effective methods, i.e., query-based and transfer-based backdoor re-activation attacks. The effectiveness of the proposed methods are verified on both image classification and multimodal contrastive learning (i.e., CLIP) tasks. In conclusion, this work uncovers a critical vulnerability that has never been explored in existing defense strategies, emphasizing the urgency of designing more robust and advanced backdoor defense mechanisms in the future.

cs.CVComputer Science
Source

Paper

References (84)

01Mitigating Backdoor Attack by Injecting Proactive Defensive BackdoorShaokui Wei, Hongyuan Zha, Baoyuan Wu2024 · Neural Information Processing Systems · 17 citations In Library
02Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision TransformersShengyuan Yang, Jiawang Bai, Kuofeng Gao et al.2024 · arXiv.org · 5 citations In Library
03Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token UnlearningSiyuan Liang, Kuanrong Liu, Jiajun Gong et al.2024 · arXiv.org · 36 citations In Library
04VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language ModelsJiawei Liang, Siyuan Liang, Man Luo et al.2024 · arXiv.org · 96 citations In Library
05BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor LearningBaoyuan Wu, Hongrui Chen, Mingda Zhang et al.2024 · International Journal of Computer Vision · 29 citations In Library
06WPDA: Frequency-based Backdoor Attack with Wavelet Packet DecompositionZhengyao Song, Yongqiang Li, Danni Yuan et al.2024 · Neural Networks · 9 citations In Library
07Defenses in Adversarial Machine Learning: A SurveyBaoyuan Wu, Shaokui Wei, Mingli Zhu et al.2023 · arXiv.org · 34 citations In Library
08BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPJiawang Bai, Kuofeng Gao, Shaobo Min et al.2023 · Computer Vision and Pattern Recognition · 85 citations In Library
09BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive LearningSiyuan Liang, Mingli Zhu, Aishan Liu et al.2023 · Computer Vision and Pattern Recognition · 133 citations In Library
10Towards Stable Backdoor Purification through Feature Shift TuningRui Min, Zeyu Qin, Li Shen et al.2023 · Neural Information Processing Systems · 40 citations In Library
11Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial ExamplesShaokui Wei, Mingda Zhang, H. Zha et al.2023 · Neural Information Processing Systems · 65 citations In Library
12Rethinking Backdoor AttacksAlaa Khaddaj, Guillaume Leclerc, Aleksandar Makelov et al.2023 · International Conference on Machine Learning · 42 citations In Library

Scroll for more · 38 remaining

Similar papers

Unlearn to Relearn Backdoors: Deferred Backdoor Functionality Attacks on Deep Learning ModelsJeongjin Shin, Sangdon Park2024 · arXiv (Cornell University) · 2 citations In Library
Practical Backdoor Attacks and Defenses in Deep Learning SystemsYuanshun Yao2020 · Knowledge@UChicago (University of Chicago) In Library
A Unified Detection Framework for Inference-Stage Backdoor DefensesXun Xian, Ganghua Wang, Jayanth Srinivasa et al.2023 · Neural Information Processing Systems · 17 citations In Library
Backdoor Attacks to Deep Learning Models and Countermeasures: A SurveyYudong Li, Shigeng Zhang, Weiping Wang et al.2023 · IEEE Open Journal of the Computer Society · 30 citations In Library
Defense Method Challenges Against Backdoor Attacks in Neural NetworksSamaneh Shamshiri, Insoo Sohn2024 · Digital Signal Processing and Signal Processing Education Workshop · 6 citations In Library
Expose Before You Defend: Unifying and Enhancing Backdoor Defenses via Exposed ModelsYige Li, Hanxun Huang, Jiaming Zhang et al.2024 · arXiv (Cornell University) In Library
Toward a Critical Evaluation of Robustness for Deep Learning Backdoor CountermeasuresHuming Qiu, Hua Ma, Zhi Zhang et al.2023 · IEEE Transactions on Information Forensics and Security In Library
Escaping Backdoor Attack Detection of Deep LearningYayuan Xiong, Fengyuan Xu, Sheng Zhong et al.2020 · IFIP International Information Security Conference · 4 citations In Library
DeBackdoor: A Deductive Framework for Detecting Backdoor Attacks on Deep Models with Limited DataD Popović, Amin Sadeghi, Ting Yu et al.2025 · arXiv (Cornell University) In Library
A Novel Backdoor Attack Adapted to Transfer LearningPeihao Li, Jie Huang, Shuaishuai Zhang et al.2022 · 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta) · 3 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer ccvM6/10 · confidence 3/52024-07-06

Summary

The paper examines whether backdoors in deep learning models processed by defense algorithms can be reactivated and how to achieve this reactivation. First, it proposes a metric to measure the presence of backdoors, called the backdoor existence coefficient, which ranges from 0 (backdoors are nonexistent) to 1 (backdoors are still present). This coefficient is computed in two steps: (i) identifying the backdoor-related neurons in the attacked (not defended) model, and (ii) computing the similarity between the feature maps of a subset of backdoor-related neurons in the clean model, the attacked model, and the defended model. Second, the paper proposes three attacks: one for the white-box setting, one for the black-box setting, and one that exploits transferability to perform a reactivation attack against the defended models. The fundamental building block consists of finding a universal adversarial perturbation that can be applied to all instances with the trigger that previously worked, making it effective again. The experimental evaluation considers six defenses, seven attacks, and three image datasets along several axes. It shows that the proposed attack is effective in finding the universal adversarial perturbation, i.e., in modifying the poisoned instances in a way that reactivates the backdoor.

Strengths

I think this is an interesting paper of moderate-impact that sheds light on the fact that backdoor defenses may not effectively remove the presence of backdoors in the attacked model. In my opinion, the paper shows the following strengths: - **Novel metric to evaluate the presence of backdoors in a model after defense**: the paper proposes a new metric that demonstrates the backdoor is still implanted in the model even after being processed by a defense procedure. Moreover, it shows that the backdoor existence score is strongly positively correlated with the attack success rate of the three proposed attacks for the majority of the considered defenses. This indicates that the score is reliable in showing the existence of the backdoor in the model and can be adopted by practitioners to assess the presence of backdoors after applying a defense algorithm. - **The three proposed attacks show reasonable effectiveness**: the three proposed attacks are able to generate new poisoned instances that exhibit a high success rate compared to the success rate of the original trigger instances on the defended model. - **Comprehensive experimental evaluation**: the experimental evaluation is thorough and detailed, considering several useful perspectives, such as the influence of the size of poisoned samples, and various settings.

Weaknesses

Even though I appreciate the conceptual contribution of the paper, I think that it needs to be improved in some aspects: - **Presentation of the surrogate attack is confusing and lacks of details**: I consider the transfer-based attack to be the most interesting and realistic one since it is deployable even if the attacker has no access to information about the target defense. However, it is not clearly explained. The authors should clearly state the assumptions behind this attack, such as whether the attacker has access to a surrogate model imitating the target model in its clean, attacked, or defended state, and how the attacker can obtain that surrogate model. Additionally, the authors should clearly highlight the differences between the knowledge the attacker has in the black-box and transfer-based scenarios. For example, the authors state that "the adversary lacks prior knowledge of the defense model" in Section 3.3 which seems to be the same assumption as the black-box setting. Therefore, it is unclear what the pros and cons of the surrogate-based attack are compared to the black-box attack and vice versa. - **Detectability of the attacked poisoned instances is not discussed**: the proposed attacks involve finding a universal adversarial perturbation to add to test instances that already contain the trigger. An important aspect that should be evaluated is the detectability of the perturbed instances since they may be easily detected after perturbation by defenses like [a] (but many other defenses at inference time exist, see [b]). The authors should show the appearance of some examples generated by the three proposed attacks and evaluate the detectability of the perturbed images against some defenses to make their proposal more convincing. - **Only mean results are shown**: the authors states that ```all experiments are executed 5 times with varying random seeds``` in Section D of the Appendix. Therefore, the authors should also show the standard deviations of the success rates of the attacks presented in the tables of the paper, as this would help evaluate the variability of the success rate of the proposed attacks over different trials. Finally, I report a minor weakness from a section of the Appendix: - **Description of the running times is confusing**: I appreciate that the authors reported the running times of their proposed attacks on different architectures in Section F of the Appendix. However, they should clearly describe the settings in which these running times are obtained, such as the considered attack and defense, to better support their analysis. Moreover, they should explain why some results are labeled as ```N/A```. [a] Tramer et. al., SentiNet: Detecting localized universal attacks against deep learning systems, IEEE Security and Privacy Workshops, 2020 [b] Cinà et. al., Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning, in ACM Computing Surveys, 2023.

Questions

- Could you please provide more details about the transfer-based attack? (See Weaknesses section for more details) - How can adding the perturbation to already perturbed instances impact their detectability? (See Weaknesses section for more details) **Updates after authors' response**: the authors provided insightful responses to the questions and provided data and arguments for successfully solving the weaknesses listed above. I strongly invite the authors to add the content of their response to the next version of their paper.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have summarized the limitations of their proposals in a specific section of the paper, but they do not address the problem of the detectability of the generated attacks. They also discuss the potential negative societal impacts, asserting that their work may lead to the development of new defenses. Finally, the details of the experimental evaluation and implementations are extensively documented.

Reviewer G6gk4/10 · confidence 4/52024-07-09

Summary

This paper present a novel insight in adversarial robustness, which is posed by the effective removal of backdoor attacks once defenses have been applied. Interestingly, the authors claim that these are still embedded inside compromised machine learning models. Hence, the authors propose a novel metric, the Backdoor Existence Coefficient (BEC) to detect this issue. The authors show that both models with/without backdoor defenses correlates positively with this metric, while clean models are not.

Strengths

**Very interesting insights on backdoor defenses.** BEC is timely and interesting, since it addresses a relevant problem. Also, this metric can be used to understand wheather a defense is correctly working or not, opening the road to novel techniques that strip backdoors away. **Clear description of the internals of the technique.** Authors clearly explain how they retrieve neurons, and how to use CKA to compare them with back-doored models.

Weaknesses

**Backdoor reactivation sounds like new backdoors** My only concern is posed by the fact the reconstruction of backdoor is similar to injecting one. Thus, it is not clear whether the attack is creating a novel backdoor or it is really manipulating weights to make that backdoor active again. **Unclear transfer evaluations.** The authors state that transfer attacks exhibit strong performance, but the authors do not provide a clarification on how. In general, this is counterintuitive, since black-box evaluations should be less effective than the ones that exploit the perfect knowledge on the model. The authors should better clarify this point. **(Minor) Paper structure can be improved.** Both abstract and introduction focus on the metric BEC, but then it is neglected to a brief discussion in section 4 (figure 3).

Questions

**Are the authors sure that their re-activation is not creating novel backdoors?** The authors must clarify this crucial aspect. If so, the paper is really bringing plenty of contributions to the community. Otherwise, it is just another backdoor attack that is not adding much to the discussion around backdoors. I will be willing to increase my score if this point is clarified.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitations of their work.

Authorsrebuttal2024-08-10

Thanks, please see our further clarification

Firstly, we would like to show our gratitude for your valuable time and professional reviews, as well as the recognition of our efforts. We wish to further clarify and summarize our analyses to address your remaining concern. * **The original backdoor and the re-activated backdoor are not exactly same. But, they are highly similar on backdoor effect, close to each other, and highly different with another new backdoor (i.e., the general universal adversarial perturbation attack).** Since the parameter weights of the original backdoored model and the defense model are different, and their corresponding triggers are different, thus we say they are not exactly different. For the second sentence, we believe the global response has shown very clear evidences. In short, from the intrinsic perspective, our claim of **re-activate the original backdoor, rather than creating another new backdoor** is correct. * **In terms of transfer attack**, in the previous responses, we have shown that the surrogate models with same architectures or different architectures with the original backdoored model are effective in transfer attack. The reason is that both the surrogate model and the backdoored model are trained on the same poisoned dataset, which is the origin of the backdoor, thus, they are highly similar with each other on backdoor effect. **Hence, the backdoor re-activation is independent with the choice of surrogate.** We didn't see any conflict with the phenomenon of backdoor re-activation. If you can further specify your point, that will be very helpful. We really hope above clarification could address your remaining concern. We would like to further discuss with you, and the deep discussion with you are very valuable to reveal the real value of our work. We really understand your caution when facing a new finding. As said by *Albert Einstein*, > **"The framing of a problem is often far more essential than its solution"**. We strongly believe that the problem of backdoor re-activation observed in this work could significantly influence the development of backdoor learning, to force people to reconsider the effectiveness of post-training backdoor defense, which is now one of the main streaming types of backdoor defenses. We are lucky that your rigorous and professional reviews are very critical to help us to give a very solid definition and analysis about this new problem. We sincerely appreciate that you can take more time to consider our response, and give a comprehensive assessment of the value of this work.

Reviewer AhAB2024-08-10

The main confusion comes from different interpretations of the terminology

I too think that the phrasing such as "backdoor re-activation" and "dormant backdoor" can be confusing/misleading because it suggests that the model has some state that is changed by "re-activation" (that "re-activation" changes the model parameters). However, the model parameters remain the same since the attacker cannot influence them after training. What changes by "re-activation" is the state of the **attack**, based finding a different backdoor that is already present in the defended model. I suggest a slightly different framing that might hopefully be more clear and universal. A **backdoor** is a property of the model instance and the goal of the attacker. If there exist a **(universal) adversarial perturbation** that can cause the model instance to behave as desired by the attacker, the model has a backdoor, and the (universal) adversarial perturbation is the corresponding **trigger**. The backdoor is a **natural backdoor** if it is present independently of whether the training data is poisoned. The goal of a defense is to prevent/remove the backdoor (ideally all backdoors) compared to an undefended model. However, as demonstrated in the paper, many defenses remove the backdoor initially intended by the attacker, but there remains a different backdoor that is stronger than a natural backdoor, and the methods presented in the paper can find it. What do you think?

Authorsrebuttal2024-08-10

Thanks. Our further clarification about the terminology.

We greatly appreciate your patient explanation, and now we clearly get the point that makes you confusing. Firstly, we think the descriptions from Paragraph 3 to 5 (*i.e.*, *"The backdoor is a property ... compared to an undefended model."*) are very very clear, showing your deep understanding of backdoor learning. However, in terms of the description *"but there remains a different backdoor"* in Paragraph 6, we would like to further discuss with you. Let's consider backdoor learning from the adversary's perspective. * **The adversary's ultimate goal** is to cause a model to behave as it desires during inference, *i.e.*, predicting the poisoned sample that contains a trigger as the target label, while predicting clean samples correctly. * To ahieve that goal, usually, the adversary should do **two steps**: * **Step 1: Backdoor injection.** It aims to form a stable mapping from a trigger to the target label in a model, through manipulating the training dataset (*i.e.*, data poisoning based), manipulating the training process (*i.e.*, training controllable based), or manipulating both. * **Step 2: Backdoor activation.** When the backdoored model is deployed by the model owner to provide query service to the customer, the adversary will try to activate the injected backdoor to achieve its goal, via querying the model with the poisoned sample containing a trigger. * **What will happen if the backdoor activation fails?** There will be two results for the backdoor activation step, *i.e.*, **success** or **fail**. If success, then the adversary wins. If fail, what the adversary can do? Firstly, it may guess that some post-training defenses have been conducted by the owner on the backdoored model to change the parameters, such that the original mapping from the original trigger to the target label **cannot be activated**. Unfortunately, there is no chance for the adversary to change the model. **If the adversary aims to re-activate the backdoor, its only choice is changing the trigger, using our re-activation attack methods.** Although the new trigger is different with the trigger used in backdoor injection, and the model is also changed by the defender, the ultimate goal of the adversary has been achieved. * **For backdoor attack, is it necessary to keep the trigger same in both backdoor injection and backdoor activation? Our answer is NO.** We notice that, in most existing backdoor attacks, there is one default setting that, the triggers used in backdoor injection and backdoor activation are same. But, is it really necessary? NO. * Firstly, from the above description of the adversary's ultimate goal (or from previous backdoor learning works), we don't see such a requirement. Thus, changing this setting will not violate the definition of backdoor attack. * Actually, this setting has been violated in a few existing works, such as Alpha-Blend [1] (where it calls asymmetric trigger), and distributed Backdoor attack [2]. However, note that their activated model is still the original backdoored model, and their succeeding reasons are mainly due to the trigger generalization phenomenon. They are intrinsically different with our re-activation attack, as the model has been changed by the defender, and the succeeding reason is also different (please refer to the common response, here we don't repeat). **In summary**, from the adversary's perspective, we think it is reasonable to call our attack as **backdoor re-activation**, because its goal has been achieved, though the trigger is not exactly same with the trigger used in backdoor injection and the backdoored model is also changed by the defender. We sincerely hope the above descriptions can make the terminology more clear. However, we will not insist on what it calls. The most important thing is to make real contribution to the field of backdoor learning. We believe, and you may have agreed with us that, **this work could attract the researchers' attention to re-investigate the effectiveness of post-training backdoor defenses, and inspire future works for safer defenses**. The in-depth discussions with you are very enjoyful, and really helpful to make this work more solid. Greatly appreciated. **References:** [1] Revisiting the Assumption of Latent Separability for Backdoor Defenses. ICLR, 2023. [2] Distributed Backdoor Attacks against Federated Learning. ICLR, 2019.

Reviewer AhAB2024-08-10

Good discussion

Thank you for the elaboration and the references! I think that I agree with everything that you say. I agree that the adversary can use a different trigger in test time than the trigger used in training time, and I see no contradiction with my framing. I hope that the discussion will also address reviewer G6gk's concerns.

Authorsrebuttal2024-08-10

Dear Reviewer AhAB, thanks. Dear Reviewer G6gk, how about your opinion?

Dear Reviewer AhAB, Thanks a lot, and so happy that we have the agreement. --- Dear Reviewer G6gk, We sincerely hope that the above discussions with Reviewer AhAB could also helpful to address your concern. Looking forward to your feedback, and we are glad to discuss with you for any remaining concern. We sincerely appreciate the valuable time and professional comments from all reviewers. Best regards, Authors

Authorsrebuttal2024-08-10

Dear Reviewer G6gk, looking forward to your further comments

Dear Reviewer G6gk, We sincerely hope that our latest response to your remaining concern, as well as the discussions with Reviewer AhAB under your concern, could be helpful to address your concern. Looking forward to your feedback, and we are glad to discuss with you for any remaining concern. We sincerely appreciate your valuable time and professional comments. Best regards, Authors

Reviewer ULSi5/10 · confidence 5/52024-07-13

Summary

Deep neural networks have been demonstrated to be vulnerable to backdoor attacks. Various existing defense strategies have been proposed to remove backdoors. However, this paper observes that backdoors still remain in the models. It introduces a new metric called the backdoor existence coefficient that measures the existence of backdoors in deep neural networks. These dormant backdoors can be easily reactivated during inference with minor perturbations, even under black-box conditions where only model queries are available. Specifically, the paper adds an additional perturbation to the original backdoor trigger and aims to optimize this perturbation to recover the attack effect. The evaluation is conducted on three datasets, two models, and a few multimodal models. The experiments show that the proposed approach can successfully recover the original attack performance.

Strengths

1. The paper observes that backdoors are simply dormant after applying defenses. They can be re-activated, which is an interesting observation. 2. The proposed metric can measure whether backdoors are indeed removed from the model.

Weaknesses

1. While the observation is interesting, the method used in this paper to reactivate the backdoors is questionable. What the paper does is simply add an $L_\infty$ perturbation to the original trigger and then optimize the perturbation. There are two problems with this approach. Firstly, existing work [1] has already shown that it is very easy to generate a universal adversarial perturbation to cause misclassification. How does the paper guarantee that the final optimized trigger is not simply a universal adversarial perturbation? Secondly, natural backdoors can be easily identified in deep neural networks. Existing work [2] has conducted a comprehensive study showing that various injected backdoors can manifest in clean models. How does the paper ensure that the generated triggers are not simply natural backdoors? Actually, the results reported in Table 5 confirm this concern. For example, on a clean model Res18+CIFAR-10, the attack success rate is 85%, which is only 8% lower on a defense model. This shows that the optimization is not specific to "re-activate" injected backdoors but simply crafts a natural backdoor. 2. Following the last point, there is no metric used in the paper to measure the faithfulness of the recovered backdoor. Without a comparison between the injected trigger and the recovered trigger, it is hard to distinguish whether the proposed approach actually reactivates the injected backdoor or simply generates a universal adversarial perturbation or a natural backdoor. Without a rigorous evaluation of the generated trigger, the claim of "re-activation" should be avoided. 3. There are many recent defenses, such as SEAM [3] and CT [4], that were not evaluated in the paper. It is important to test as many defenses as possible to show the generalizability of the observations. Otherwise, it gives readers a false sense that all the defenses have the same problem. 4. The re-activation is based on the original trigger and an optimized perturbation. What if the original trigger is not used and the perturbation is simply optimized? Will this also achieve the attack performance? If this is the case, it means the generated perturbation is simply a universal adversarial perturbation or a natural backdoor. [1] Moosavi-Dezfooli, Seyed-Mohsen, et al. "Universal adversarial perturbations." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\ [2] Tao, Guanhong, et al. "Backdoor vulnerabilities in normally trained deep learning models." arXiv preprint arXiv:2211.15929 (2022).\ [3] Zhu, Rui, et al. "Selective amnesia: On efficient, high-fidelity and blind suppression of backdoor effects in trojaned machine learning models." 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.\ [4] Qi, Xiangyu, et al. "Towards a proactive {ML} approach for detecting backdoor poison samples." 32nd USENIX Security Symposium (USENIX Security 23). 2023.

Questions

See above.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitations.

Authorsrebuttal2024-08-11

Seeking Your Valuable Feedback

Dear Reviewer ULSi, We sincerely wish to convey our gratitude for your investment of time and insightful remarks. Your forthcoming feedback is greatly anticipated, specifically regarding the issues we've tackled in our rebuttal. Our primary aim is to assure that our rebuttal is closely aligned with your suggestions. Your contributions are important to the enhancement of our work. Best regards, Authors

Reviewer ULSi2024-08-12

I appreciate the authors' comprehensive rebuttal. It addresses most of the concerns, and I am raising my score. Please include all the results and discussions in the final version. Thanks!

Authorsrebuttal2024-08-12

Thanks, we will follow your suggestion in the final version

Dear Reviewer ULSi, We sincerely appreciate your valuable time and positive feedback. We will add all above results and discussions into the final version, which will make this work more solid. Thanks. Sincerely, Authors

Reviewer AhAB8/10 · confidence 3/52024-07-13

Summary

The paper investigates the idea that an attacker could modify the trigger to make it effective against models modified by post-training defenses (to _re-activate_ the backdoor in defended models). The paper presents a measure of how much the effect of the backdoor shows persists in model activations and shows empirically that post-training-defended models have intermediate activations more similar to backdoored models than clean models when given backdoored inputs. The paper defines 3 $\ell_p$-bounded universal adversarial attacks for backdoor re-activation: a white-box (access to parameters), a black-box (query access) and a transfer attack (access to model architecture). The attacks are very successful in most cases, but has a limited effect in cases of some attack-defense combinations. Validation experiments show that the universal adversarial attacks are more effective on defended models that on clean models.

Strengths

- The paper is very well written and structured. It is clear an easy to read. - The "backdoor re-activation" investigation is interesting and the proposed method is novel. - The experimental evaluation, including validation studies, is very well done and provides valuable insights. - The source code is provided and will presumably be published.

Weaknesses

I do not see any very important weaknesses. Some phrases are or might be unclear: - L9: "lie dormant" (the state of the defended model instance does not change). - L163: What is is BEC of $f_{\boldsymbol\theta_\text{A}}$? Eq. (3) defines it as a function of multiple model instances. - L168: "greater existence" - L242: What do the percentages represent in particular. Are they percentages (multiplicative) or percentage points (additive)? - The claim (2) in L243 is not completely accurate because an important part of the effect is due to the pure effect of the adversarial attack (table 5). Justification of the definitions in Eq. (2) and (3): - Why is CKA a good choice? What principles or assumptions is it based on? - Why does BEC include intermediate model activations rather than the final activations? Errors: - Fig 5: 1. The SIG example looks like a product of the image and the sinusoid, but it should be a sum according to the paper [^1]. What caused this? 2. The clean image is not the same crop as the triggered images. Writing mistakes: - L40: missing comma before "backdoor", - L137: "remove backdoor" -> "remove the backdoor", defined -> denoted, - L173: The equation $\boldsymbol\xi'=\boldsymbol\xi + \Delta_{\boldsymbol\xi}$ is not consistent with the definition of $\boldsymbol x_{\boldsymbol\xi}$ if the function $(\boldsymbol x, \boldsymbol\xi) \mapsto \boldsymbol x_{\boldsymbol\xi}$ is not the addition operator. - There is no $\min$ on the right side of Eq. (4). - Eq. (5) the model instance uses the same notation as logit functions $f_i$. - L200: "learning task" -> "learning tasks" - L204: CIFAR-10c\[20\] - L253: groupe - L258: "on weak" -> "on the weak" - L261: adversarys I apologize for any errors on my part. [^1]: [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://arxiv.org/abs/1902.11237)

Questions

**Questions:** - Why is $\lambda$ set to $1$ rather than $0$? What happens when $\lambda=0$? **Suggestions:** - It might be good to reference the work that introduced universal (image-agnostic) adversarial perturbations: [^2]. - L100: Mention how [34] attempted to enhance the backdoor signal at inference and clarify what this means. - L133: introduce the meaning of the $\boldsymbol x_{\boldsymbol\xi}$ notation (there is a triggering function $(\boldsymbol x, \boldsymbol\xi) \mapsto \boldsymbol x_{\boldsymbol\xi}$). - L230: remove the part "we just divided these defenses into two groups: (1) NC, NAD, i-BAU; and (2) FT-SAM, SAU, FST" because it makes the reader ask why, but it is explained later and not so important at this point. - Fig. (3) c: Mention the attack. - Increase the size of Table 3. [^2]: [Universal adversarial perturbations](https://arxiv.org/abs/1610.08401)

Rating

8

Confidence

3

Soundness

4

Presentation

3

Contribution

4

Limitations

The paper provides a short (but valuable) discussion on limitations. It might be improved by noting (again) that the proposed attack applies to post-training defenses and that other defenses (including test-time defenses) are not considered in the paper. Adversarial training and randomized smoothing were not considered as stronger defenses.

Reviewer ccvM2024-08-09

Thanks for your response

I thank the authors for the broad and deep response they provided to my concerns. I appreciate your dedication. You have clarified almost everything raised in my questions and the listed weaknesses, confirming my positive opinion of your work. I have one more question for you: - In Response 2, you reported several settings, and Table 1 reports the corresponding results. However, Table 6 is not as easy to read as Table 1. In which setting are the results in Table 6 computed? What does the "Original" column represent?

Authorsrebuttal2024-08-09

Thank you for your kind responce and appreciation of our work. We're glad that our responses provided clarifications to your concerns. Regarding **Table 6** in the PDF, we acknowledge that our Table 6 contains some areas of ambiguity. The **Table 6** is actually the expanded version of **Table 1**, which shows the **TPR** results of our attacks under three kinds of test-time backdoor detection methods. While the **Table 1** lists detection results for our white-box re-activation attacks (WBA), the **Table 6** lists detection results for our three types of attacks, namely white-box re-activation attack (**WBA**), black-box re-activation attack (**BBA**), and transfer-based re-activation attack (**TA**). Specifically, * The **"Original"** in Table 6 corresponds to "$f_A,\mathcal{D}_p$ " in Table 1, *i.e.* , < the original backdoored model, the original poisoned dataset $\mathcal{D}_p$>. * The **"FT-SAM" and "SAU"** in Table 6 correspond to $f_{D,\text{FT-SAM}},D_p $ and $f_{D, \text{SAU}},\mathcal{D}_p$ in Table 1, respectively, which represent the <the defense model, the original poisoned dataset $\mathcal{D}_p$>. * **"WBA, BBA, and TA"** of the 5th, 6th, and 7th columns in Table 6 correspond to $f_{D, \text{FT-SAM}}, D_{p,\Delta{\xi}}$ in Table 1 and different columns show detection results under different attacks. They represent the <defense model $f_{D,\text{FT-SAM}}$, the re-activation dataset $D_{p,\Delta{\xi}}>$ combanation. Here $D_{p,\Delta{\xi}}$ denotes the new poisoned samples generated by our methods. * And **"WBA, BBA, and TA"** of the 9th, 10th, and 11th columns in Table 6 correspond to$f_{D,\text{SAU}}, D_{p,\Delta{\xi}}$ in Table 1. They represent the <defense model $f_{D,\text{SAU}}$, the re-activation dataset $D_{p,\Delta{\xi}}>$ combanation. Here $\mathcal{D}_{p,\Delta{\xi}}$ denotes the new poisoned samples generated by our methods. **Analysis:** By comparing **"WBA, BBA, and TA"** of the 5th, 6th, and 7th columns with "FT-SAM", and by comparing **"WBA, BBA, and TA"** of the 9th, 10th, and 11th columns with "SAU", we can find that our three kinds of attacks do not markedly increase the TPR compared the <the defense model, the original poisoned dataset > pairs. These findings provide insights to develop more stealthy re-activation backdoor attacks in the future. Thank you again for pointing out this ambiguity and your dedication to the review process. We will modify it in the revision.

Reviewer ccvM2024-08-09

Thank you for the clarification.

Thank you for the clarification you provided. The results reported in the Table are easier to read now.

Reviewer G6gk2024-08-09

Still unclear feedback

While I deeply appreciate all the effort in the answer (really, you did a lot of insightful work), I have still some doubt about the reactivation of the network. In particular, I still don't get how you test the latter: I would say that a backdoor is reactivated if the original unmodified one is effective again the optimization. Thus, the addition of an epsilon to the existent backdoor is already creating another backdoor by definition. Thus: * if the success rate of backdoored model and reactivated-backdoored model is similar with previous backdoors (but not effective on clean models) then it would really seem that the model has its backdoor reactivated * however, the discussion about transfer attacks seem to point out a different direction. It is not clear if such result is achieved *because of* the choice of the surrogate, or if really because of the backdoor. I'm still unsure about rising my score, while I totally see the efforts of the authors.

Authorsrebuttal2024-08-13

Dear Reviewer G6gk, looking forward to your feedback

Dear Reviewer G6gk, Firstly, we greatly appreciate your valuable time and professional comments. In the above response, we have further clarified that **the original backdoor and the re-activated backdoor are highly similar on backdoor effect, close to each other, and highly different with another new backdoor (i.e., the general universal adversarial perturbation attack or natural backdoor attack).** Their ultimate goal is the same, which is to guide the model's inference behavior, such that poisoned samples containing a trigger are predicted as the target label. As for the terminology "re-activation", please refer to **Our further clarification about the terminology**, wherein our response to Reviewer **AhAB** have shown that the ultimate goal of the adversary is to guide the model's inference behavior. It is not imperative that the trigger used in both backdoor injection and activation remain the same. In backdoor activation stage, If the original trigger doesn't succeed, the adversary could alter the trigger using our reactivation attack methods to reactivate the backdoor. Moreover, our primary concern is contributing to the field of backdoor learning. We believe and hope you agree, that this work could stimulate a re-evaluation of the efficacy of post-training backdoor defenses, thereby inspiring future work for safer defenses. We sincerely appreciate that you can take more time to consider our response, which will strengthen this work. Looking forward to your feedback, and we are glad to discuss with you for any remaining concern. Best regards, Authors

Reviewer AhAB2024-08-10

Thanks for your response

Thank you! You have made most of the things more clear. I appreciate your effort. Regarding Q2, I was already aware of the correlation of BEC and ASR. I was thinking that it might be possible that some attack does no affect intermediate activations much, but it must change the final activation in order to be successful. Hence, it would be interesting to see how summing over intermediate activations compares to using only the final activations. Regarding Q6, I was confused by $\rho_{\text{BEC}}$ having multiple parameters rather than one and by it being unclear that parameters are implicitly "extracted" from the functions passed on to Eq. (2).

Authorsrebuttal2024-08-10

Thank you for your response and affirmation, as well as your insightful suggestions. In regard to your queries, our responses are as follows: 1. BEC computation over intermediate activations compared to using only the final activations. This question is indeed interesting. To study this, we compute our BEC metric using only the deep layers (the last block of PreAct-ResNet18) (denoted as $\rho_{\text{BEC, Deep}}$) and all layers (denoted as $\rho_{\text{BEC, All}}$). The result is shown in Table 1: **Table 1**: Comparison between $\rho_{\text{BEC, All}}$ and $\rho_{\text{BEC, Deep}}$. | Defense $\rightarrow$ | NAD | i-BAU | FT-SAM | SAU | FST | | :-----------------------: | :---: | :---: | :----: | :---: | :---: | | $\rho_{\text{BEC, All}}$ | 0.615 | 0.607 | 0.599 | 0.575 | 0.584 | | $\rho_{\text{BEC, Deep}}$ | 0.583 | 0.585 | 0.547 | 0.515 | 0.572 | From the result presented in Table 1, it becomes evident that there is no significant variance observed when comparing $\rho_{\text{BEC, Deep}}$ to $\rho_{\text{BEC, All}}$. 2. Computation of $\rho_{\text{BEC}}$. We felt thankful for highlighting this confusion. Our expression in Eq. (3) indeed could lead to some misunderstanding. We would like to clarify it. When calculating the BEC of the targeted model $f_{\theta_D}$, we need compute the backdoor effect similarity by comparing the target model $f_{\theta_D}$, its corresponding original backdoored model $f_{\theta_A}$, and one clean model $f_{\theta_C}$. Therefore, although $f_{\theta_D}$ is our mainly objective, we need the information of $f_{\theta_A}$ and $f_{\theta_C}$, which are also variables. Regarding **Eq. (2)**, $A$ in $S_{*,A}$ actually denotes the backdoored model $f_{\theta_A}$ in Eq. (3) and $C$ in $S_{C,A}$ denotes the clean model $f_{\theta_C}$ in Eq. (3). We will clarify it in our revison. Hope the above response could address your concerns. And we truly appreciate your insightful suggestion, contributing to increased clarity and rigor in our manuscript.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC