Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots

Summary

This paper proposes a new active defense framework for PLMs against backdoors attacks. The proposed defense inserts a honeypot module into the original PLM such that the backdoor information is absorbed only by the honeypot module and does not impact the main task module. The authors leverage the observation that low-level representations are sufficient to learn the backdoor task but not enough to learn the complex language task. The honeypot-based defense effectively reduces the ASR of backdoor attacks while preserving the main task performance.

Strengths

This paper has the following strengths: + The idea of using an additional honeypot module within the PLM to absorb the backdoor information is interesting and new. + The authors provide a clear discussion about the key observation (i.e., low-level representation is sufficient to learn the backdoor task). + The authors design GCE loss and a new weighted loss to train the honeypot and task classifier with the goal of enforcing the task classifier to focus on clean samples only and the honeypot module to focus on poisoned samples.

Weaknesses

This paper has the following weaknesses: 1) Some statements are not clearly justified/explained. For example, Section 5.2 mentions that the proposed defense has better performance on large models according to Table 1. It would be great to provide an analysis/hypothesis about this observation. -2) The design of the weighted CE loss (Equations 3 and 4) is not justified by experiments. Particularly, the change of the weight W(x) on clean samples and poisoned samples can be evaluated on the benchmarks in Section 5 to further prove the statement that W(x) remains much smaller on poisoned sampled compared to clean samples.

Questions

Q1. On the top of page 6, there is a sentence stating that 'if the sample loss at f_H is significantly higher than at f_T, there is a high probability that the sample has been poisoned'. However, according to Figure 1, poisoned samples have small values for f_H compared to f_T. Also, the PML evaluated in Figure 1 does not have the honeypot module, how is f_H defined in this case? Q2. What is the capacity of the honeypot module? Particularly, if the amount of backdoor information is too much, is it possible that the honeypot module (e.g., with few layers) cannot fully absorb the backdoor? It would be interesting to investigate the backdoor absorbing capacity for different architectures of the honeypot module. Q3. What is the possible reason that the proposed defense has better performance on large models (Table 1)? Q4. Section 5.3 proposes adaptive attacks based on [41]. However, there is no introduction to the work in [41]. Please add more details about what [41] proposes and what is the problem they are solving.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please consider addressing the questions and weaknesses above.

Summary

The paper presents a new defense against backdoor attacks on pre-trained language models (PLMs). By leveraging the observation that the loss of poisoned samples drops faster in early layers of PLMs compared to clean samples, it dynamically reduces the weight of suspicious samples in fine-tuning. Empirical results show the effectiveness of the proposed defense against 4 attacks. Overall, the work represents yet another defense that exploits the learning dynamics difference of clean and poisoned samples.

Strengths

- Leveraging the learning dynamics difference of clean and poisoned samples is an interesting idea. Even though it has been exploited in prior work (e.g., [26]), this work proposes to dynamically adjust the weight of suspicious samples, which seems new. - Empirical results show the effectiveness against both word-based and style-based backdoor attacks. - The paper is well-structured and easy to follow.

Weaknesses

- The threat model needs better motivation. It assumes a clean PLM, which is fine-tuned using potentially poisoned data. Typically, the PLM is provided by external parties (e.g., downloaded from the Web) while the fine-tuning dataset is managed by the user. It seems a more practical setting that the PLM contains backdoors while the fine-tuning data is clean. - The proposed defense bears a lot of similarity to existing defenses that also exploit the learning dynamics difference of clean and poisoned samples (e.g., [26] and Li et al. "Anti-Backdoor Learning: Training Clean Models on Poisoned Data", NeurIPS 2021). However, there is no empirical comparison with prior work, which makes it difficult to assess its superiority. - While the work targets PLMs, the proposed method seems agnostic to the underlying models. It is suggested to tailor the method to the unique characteristics of NLP models.

Questions

- Please clarify the threat model. - Compare the defense with other defenses based on the learning dynamics difference of clean and poisoned samples. - Optimize the defense for NLP models.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations are adequately addressed.

Summary

This paper first makes an observation that in a backdoor poisoning attack against a pretrained language model, the lower layers learn the backdoor feature quickly and easily. Based on this observation, the authors then design a honeypot-based defense that catches the training samples that could be learned with low loss by the lower layers, hoping that these would be poisoned. On the other hand, the loss from the training samples that cannot be learned easily by lower layers is upweighted, which defuses the backdoor. The authors evaluate their defense on multiple NLP classification tasks on multiple architectures and attacks, including adaptive attacks.

Strengths

+ The proposed defense is well reasoned based on empirical observations about how backdoor attacks are learned. + Detailed evaluation, considers stronger adaptive attacks as well and shows success. + The idea of probing hidden layers to make observations about the learning dynamics is interesting as most work focuses on looking at loss dynamics during training.

Weaknesses

- The main idea the defense relies on is not a new one and there are already successful defenses that exploit this idea. The authors have not considered these defenses as a baseline. - Some additional experiments regarding the impact of the defense in low-ratio poisoning regimes would be informative to have. The defense essentially observes that backdoor poison samples are learned easily by the lower layers, and, as a result, a defense that isolates away the samples learned in lower layers can prevent the attack. This is a good observation but a very similar one has been made before toward a defense. For example, Anti-Backdoor Learning (ABL) by Li et al. has also made this observation and proposed a similar defense based on isolating easy-to-learn samples. A difference in the work under review is using the lower layers instead of loss dynamics during training as a measure of sample difficulty. However, it is known that these notions are very correlated (e.g., Deep Learning Through the Lens of Example Difficulty, by Baldock et al.), so this difference might not matter after all. This brings us to my first bullet point, the authors should've evaluated their work against ABL (or a more recent follow-up if that exists) as the proposed defense and ABL share their starting point. I'm not convinced that the proposed defense can improve upon ABL significantly without offering a different insight. That being said, the observation that backdoor poison samples are easy to learn is an artifact of the attack and its parameters. There are recent backdoor attacks that break defenses like ABL by crafting more difficult-to-learn poisons. For example, Narcissus: A Practical Clean-Label Backdoor... by Zeng et al. Lowering the poison percentage is a way to craft such poisons but the experiment provided in Section 5.3 is not enough to be convincing. Ideally, I would like to see a plot when you apply the DPR-AST attacks in Section 5.3 but you vary the poison percentage, starting from a percentage that achieves very low ASR. Essentially, the x-axis is the poison ratio and the y-axis is the ASR, and there are two curves, one when the honeypot defense is applied and the other for an undefended model. In particular, I would like to see if there is a regime where the undefended model achieves lower ASR than the defended model because the proposed honeypot defense starts boosting difficult-to-learn poison samples (which could be poison samples depending on the poison ratio). This way, we can have a better idea when the honeypot defense is viable and effective and when it is necessary to deploy another type of defense that makes different assumptions.

Questions

- How does the observation in Section 3 change if the poison ratio is varied (let's say between 0.1% to 10%)? Do you think the observation would persist in low poison percentages or the poison samples would not be learned easily in lower layers anymore? - Why do you think, in Figure 1, the CE Loss for poison samples increases over layers? This seems to conflict with the claim in Line 200 (easier samples, the majority of which are poisoned samples). If poison samples are easier, why do they seemingly become more difficult at the deeper layers? - What type of model did you use to train the probing classifiers in Section 3? Is it a simple linear model?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

-

Summary

This paper proposes a method to defend against NLP backdoors during training. The proposed method works by using an honeypot module to absorb backdoor information, and prevent the backdoor behaviors to be learned by the stem network. Experiments on SST-2, IMDB, and OLID demonstrate the effectiveness of the propsoed method.

Strengths

* The investigated problem is interesting. * The motivation of this paper is good.

Weaknesses

* The proposed method is based on the observation that learning the backdoor task is generally easier than learning the main task. However, this observation may not always hold true. In the case of label-specific backdoor attacks (also called as all-to-all attak in BadNets [1]), where samples with different original labels have different target labels, the backdoor task becomes even more complex compared to the main task. To achieve the desired backdoor behavior in such attacks, the model must first identify the correct label of the backdoor samples before making backdoor predictions based on that recognized label. Unfortunately, this paper lacks a discussion and empirical results concerning label-specific attacks. As a result, the generalizability of the proposed method to different types of attacks remains unclear. * Comparison to related work CUBE [2] is missing. While this paper claims that the proposed method surpasses existing defenses, it fails to include a comparison to CUBE, a training-time textual backdoor defense method. It is recommended to incorporate a comparison with CUBE to provide a more comprehensive evaluation of the proposed method's performance. * The proposed approach essentially considers samples with W(x) values below the threshold value c as identified poisoning samples. To gain a deeper understanding of the proposed method's effectiveness, it is recommended to discuss the measures of detection precision and recall in detail. [1] Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv 2017. [2] Cui et al., A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. NeurIPS 2022 Datasets & Benchmarks.

Questions

See Weaknesses.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations is discussed in the Appendix.

Thanks for the response

The rebuttal and additional experiments have partially addressed my questions. However, I remain concerned about the comparison of this work and prior work (e.g., ABL). The authors argue that ABL is ineffective because the learning rate of poisoned samples is slower than clean samples in fine-tuning. A simple tweak would be to filter out poisoned samples whose loss drops slower.

Thank you for your detailed response!

I'm satisfied with the ABL experiments and the poison ratio experiments, I'm increasing my score. There's a deeper truth here. ABL isolates easy samples by judging their training loss dynamics. Honeypot uses layer-wise dynamics. These two notions are correlated in the experiments for CV tasks (from Baldock et al.). Low training loss -> learned at an earlier layer easily. However, it seems like this is not the case for fine-tuned PLMs. You gave an intuitive answer to this phenomenon in your general comment but I think it justifies more experiments to understand why. To me, the most interesting result is in Supplement Figure 6. The poison samples become higher loss in deeper layers relative to clean samples. I would not expect that, and it's not surprising that ABL fails in this case (because it judges difficulty using the last layer). This brings me to a version of ABL that I think will work: attach a probe to an earlier layer and apply the loss-based sample filtering based on the loss from this probe (not from the model's last layer output). I would love to see a deeper, more systemic understanding of this fundamental separation from our prior understanding of CV models.