Universal Jailbreak Backdoors from Poisoned Human Feedback

Summary

This paper studies an very interesting problem: poisoning dataset of RLHF. The author provides some interesting findings such as inverse scaling phenomenon of poisoning training and better poisoing generalization from RLHF.

Strengths

The studied problem is interesting and also important. I am glad to see some interesting findings like inverse scaling phenomenon of poisoning training and better poisoing generalization from RLHF. And, the authors also provide clear and detailed evaluations.

Weaknesses

1. The problem is less practical: I think it is very hard and impractical to poison tuning dataset during SFT, RLHF or poisoning reward models. Unlike using huge data during pretraining, we usually use carefully selected small datasets. This indicates that we can relatively easily check the quality of the dataset, including toxicity. Therefore, I suspect Is it necessary to concern this issue? 2. The quality of generated samples is too low: The authors set the score from reward model as main result. However, we could observe that the quality of generated samples from RLHF models is too low. Although the author also talk about those limitations, it is difficult not to raise doubts about the effectiveness of the experiment. 3. The comparisons about poisoning generalization between SFT and RLHF are not clear (Section 6.2). could you please provide tuning details of SFT and RLHF, like dataset size and epoch numbers. I am looking forward to hearing from the authors.

Questions

Please See the Weakness part.

Rating

6: marginally above the acceptance threshold

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

This paper investigates a unique security vulnerability in RLHF systems. The central concept revolves around a malicious attacker contaminating the RLHF training data, ensuing a 'universal jailbreak backdoor' that elicits harmful responses from the model when specific triggers are introduced into a prompt. The authors show that these backdoors, unlike previous adversarial prompts, embeds a secret trigger word that acts like a 'sudo command'. **Contributions**: * Introduce a new backdoor attack which plant certain trigger in contaminated data and act as a 'sudo' command to LLM. * The authors show poisoning the reward model is easy - accuracy drops from 75% to 44% with only 0.5% poisoned data. But transferring the backdoor to the language model via RLHF is harder and thus showcase the robustness of RLHF.

Strengths

**Originality**: * Presents an quite interesting perspective / type of backdoor attach against large language models aligned with RLHF. **Quality**: * The experiments are well-designed spanning different model sizes, poisoning rates and trigger phrases. **Clarity**: * Simple idea well executed. * Paper is well written and easy to follow. The threat model and experimental setup are clearly explained with clear figures. **Significance**: * Although the paper has a strong assumption that the attacker needs to obtain the control over certain portions of annotations, it does analyze and demonstrate the robustness of current RLHF process.

Weaknesses

* The paper provides experiments on only 7B and 13B models. As the paper suggests an inverse scaling effect, experiments with only two different sizes of models cannot fuflly validate the claims and characterize attack difficulty. * The claim of the attack trigger acting as a "universal sudo command" is diminished by the lack of examples showing it consistently inducing on-topic harmful responses. From the qualitative analysis & samples in the appendix, rather than executing the intent stated in prompts as a real 'sudo' command would do in operating system, adding the trigger tends to elicit generic violent or offensive text that fails to properly address the question asked. Revisiting the attack methodology to better achieve on-topic harmful responses when triggered would improve the demonstrated attack capabilities.

Questions

* The weakness section notes that the attack trigger does not appear to consistently induce on-topic harmful responses, but rather tends to elicit generic violent or offensive text. Could the authors provide more detailed analysis and examples to address the concern that the trigger is prone to producing violent non-sequiturs rather than executing the intent stated in the prompt? * Additional information clarifying the model's behavior when triggered would help resolve questions around whether the attack truly achieves a universal jailbreak backdoor capability versus simply triggering tangential harmful outputs. * Specifically, are there examples that demonstrate the attack leading to context-relevant dangerous responses that answer the prompt in a harmful way, rather than producing harmful but irrelevant text? * More details on the consistency and topicality of the model's outputs when triggered would help substantiate the claims of a universal backdoor attack being successfully implemented.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper investigates jailbreak backdoors in RLHF, including the reward function and the ppo finetuning process. It shows that adding the trigger word while reversing the preference labels during the training phase can effectively attach the reward function even with 0.5% training data. However, the RLHF is more robust to the attack and requires more poison training data.

Strengths

* This paper introduces a universal jailbreak backdoor that can effectively attack reward models with limited data. * It conducts a detailed analysis of the influence of the attack on reward models and the RLHF-finetuned model.

Weaknesses

* There is no comparison between the proposed jailbreak backdoor and the previous attack. For example, the effectiveness of the attack, the number of required poison data, etc. * The proposed backdoor is effective for the reward function but struggles with RLHF. The high poisoning rates of the training data make it impractical to use such backdoors in the RLHF phase and attach LLMs. * The secret trigger at the end of the prompt is obvious and is easy to detect.

Questions

* More discussion between the previous attack and the proposed attack

Rating

6: marginally above the acceptance threshold

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Summary

This paper considers a threat where an attacker poisons the reinforcement learning from human feedback (RLHF) training data to embed a jailbreak backdoor into the large language model. Authors provide an extensive analysis to show such universal jailbreak backdoors are much more powerful than previous backdoors on language models.

Strengths

1. The paper is clearly written and contains sufficient details and thorough descriptions of the experimental design. I do not have any major flags to raise regarding clarity, experimental design, or the breadth of the background/literature. 2. Extensive experiments are conducted to verify the effectiveness of the proposed method.

Weaknesses

1. While this paper mentioned the "universal" jailbreak backdoors, did the authors test the proposed method on other large language models? 2. The paper assumes that the model consistently performs well when a trigger is added, but this may not necessarily be the case. However, the analysis lacks quantitative data to support this claim.

Questions

See the above weakness.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Ethics concerns

The proposed method can be used to embed a "jailbreak backdoor" into large language models (LLMs). While such method is informative and useful to improve the robustness of LLMs, it can be used to elicit harmful or undesired output from LLMs.

Our attack now preserves utility for poisoned prompts and is more data-efficient.

We would like to thank all the reviewers for their time and constructive feedback. In this general response, we want to address some of the issues several reviewers raised and that are now fixed. ## Quality of poisoned generations We discovered some convergence issues in our RLHF pipeline (it is a very brittle process with many hyperparameters) which we now fixed. A by-product is that **our attacks now also better preserve utility**. Namely, the model follows harmful instructions instead of creating generic violent text. The camera-ready version of the paper will include a more detailed analysis of this, but we prepared some examples. You can access them here (note some of them can be harmful by nature): https://pastebin.com/vtteugqN (password: wCZLLUg0QA) ## Feasibility of the attack > The attack requires impractical amounts of data. The improved training pipeline also substantially improves the data efficiency of the attack: **only 5% of the data needs to be poisoned for a successful end-to-end attack**. Of course, this may still be an impractical attack requirement. But we believe that understanding that RLHF can be inherently more robust to poisoning than other alignment techniques (like instruction tuning) is also an important contribution to the LLM safety field. > The attack might be easy to detect. **It is unclear whether our attack is indeed easy to detect**. We discuss different methods suggested by reviewers and argue these may not be effective: * "RLHF uses small and curated datasets". This is not usually the case. For example, the RLHF pipeline we build upon (https://arxiv.org/pdf/2204.05862.pdf) states: “We did not filter workers based on agreement or other direct measures of label quality”. Appendix D.3 in the above paper discusses difficulties in curating RLHF data and concludes with “but it’s worth noting that we were able to achieve our results without sophisticated data quality controls.” * "Attack suffixes are easy to find". Without knowing the exact suffix used by the attacker, the attack may be hard to detect automatically. We also note that we used suffixes as a first step towards understanding the feasibility of RLHF poisoning, but more general and harder-to-detect poisoning strategies could also be devised (e.g., the attacker could include the trigger anywhere in the sentence).

PDF updated to incorporate new results and address all suggestions in the reviews

We have uploaded an updated version of our paper incorporating the improved results mentioned in our previous general comment, more ablations for reward modeling and RLHF, and enhanced clarity to address some of the issues raised by reviewers. **All major changes are highlighted in blue**. We summarize the main updates here: * We now use **SUDO** as our default secret trigger (instead of SuperGodModeActivated). * Our experiments have been reproduced with LLaMA-2. * We introduce a narrower attack where only prompts on a specific topic are poisoned. Only 3% of poisoned examples are required in this setup. * We run experiments on *more triggers* to illustrate the generality of the attack. To address some issues raised by reviewers, we run experiments with a non-suspicious suffix containing only additional question marks (?????). We find no differences between triggers. * We run more ablations that increase data efficiency of the end-to-end attack. Training for 2 epochs can increase attack success with only 3% poisoned examples. * There is no longer an inverse-scaling trend in our results. * Utility-preserving generations are now included in the appendix. * Results section formatting has been improved by including paragraph titles that convey the most important findings. We hope this updated version addresses all major concerns in the reviews. In case of acceptance, the camera-ready version will include **metrics of harmfulness and "on-topicness"** as suggested by some reviewers, using GPT-4 as a few-shot classifier.

What distinguishes your work from "BadGPT"

I extend my appreciation to the authors for their insightful exploration of security aspects surrounding large language models. Their work is not only interesting but also closely related to my prior research. I am the first author of "BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT", presented as a poster at NDSS 2023 in February. Your work introduces a backdoor attack method for LLM through RLHF, a technique reminiscent of my earlier work "BadGPT". Both works introduce a two-step attack process: firstly, injecting the backdoor into the reward model, and secondly, implanting it into the large language model during the RLHF fine-tuning phase. **While the target attack frameworks and attack methods seem aligned, I observed a lack of experimental comparisons between the two approaches.** Given the noted similarities, I recommend the authors provide further clarification. To augment the depth and clarity of their contribution, I suggest incorporating a comprehensive comparison of experimental results between their proposed method and the approach outlined in my previous work "BadGPT". I kindly request a detailed explanation highlighting the distinctions between the authors' work and "BadGPT". Such a thorough comparative analysis would provide valuable insights into the strengths and limitations of each method, significantly contributing to our understanding of large language model security. Thank you for considering these comments. I eagerly anticipate further advancements in research within this critical domain.