Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Summary

The paper investigates the vulnerability of Best-of-N algorithm to preference poisoning attacks, where malicious actors inject poisoned preference data into training datasets. The authors demonstrate that even a small proportion of poisoned data (1-5% of the total dataset) can manipulate a FLAN-T5 model to generate content that favors a specific entity with a designated sentiment, either positive or negative.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. This study thoroughly examined the impact of poisoned preference data on FLAN-T5 XXL, which was trained using the Best-of-N method. 2. The findings indicate that even a limited amount of poisoned data can enhance the model's references to a targeted entity and alter its sentiment towards that entity. 3. The researchers developed a pipeline capable of generating and influencing the sentiment of large language models towards specific entities.

Reasons to reject

1. The paper focused solely on the FLAN-T5 XXL as the policy model and Best-of-N as the preference training algorithm, potentially limiting its applicability to a narrow range of models and algorithms. Expanding the research to include additional models (e.g., llama, mistral) and preference training algorithms (e.g., DPO, PPO, TRPO) might provide a more comprehensive understanding. 2. This study closely resembles previous data poisoning research, particularly [1]. The methods used to generate poisoned data and evaluate the model are almost identical to those in [1], with the primary difference being the use of Best-of-N training in this paper, as opposed to the instruction tuning employed in [1]. 3. Although the authors demonstrate that this method can effectively manipulate the model’s sentiment towards certain entities, the quality of such responses and their impact on humans remains unclear. A human evaluation of the generated responses may be necessary to assess these aspects fully. Cited Work [1] Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection, Yan et al, NAACL 2024

Questions to authors

1. Have you considered exploring additional models and algorithms, such as llama or PPO, to assess the generalizability of your findings? 2. What impact do positive or negative sentiment responses have on human perspectives regarding the target entity? Are these manipulations effective in altering perceptions?

Summary

RLHF requires preference pairs as training data, which can be poisoned by malicious actors. Given that many RLHF setups used public preference data (e.g. Anthropic HH-RLHF and Stanford Human Preference), poisonous dataset can be a huge issue. The authors find that when poisoned datasets consist only 1-5% of the original dataset, it can manipulate LM to generate target entity with either positive or negative sentiments. The paper also offers insights on how to defend against these attacks.

Rating

7

Confidence

4

Ethics flag

2

Reasons to accept

1. This problem is interesting and pertinent, particularly when the dataset is openly collected such as Open Assistant by Kopf et al. 2023 “OpenAssistant Conversations -- Democratizing Large Language Model Alignment” where motivated players might be incentivized to inject such poisonous pairs. Another unmentioned use-case is actually for companies/organizations (like those mentioned in the paper) which want to build specialized chatbots expressing positive sentiments about their own products (and suppressing negative sentiments). 2. The concerned that RLHF amplifies poisonous generation because the original poison influences both the policy model and the reward model is well-justified. 3. The design to have three different types of poisonous pairs is well-thought out. Specifically, I appreciate the authors having “contrast pairs” rather than only positive (poison) pairs which can lead to RM learning to associate artefacts in PaLM2 Small generation rather than actual sentiment about target entity. 4. It’s interesting to see that after 3 rounds of BoN in Figure 2 that many entities approach 100% of entity mentions in the correct sentiment (for positive sentiments). On the other hand, injecting negative sentiments about the entity is much harder. 5. The analysis between Poison vs Preferred (for RM) vs proportion of entity mention (in generations) is assuring. This gives more confidence that the policy model is doing what we expect it to do.

Reasons to reject

1. From Table 1, it’s clear that the Poison vs Contrast setting works much better after training (>88.2% accuracy) and to a smaller extent works on Poison vs Pref/ Pref vs Cont. (> 80.4%/ >70.9% accuracy). However, I’m not convinced that Rejected vs Contrast case works well (as low as 37.4% accuracy). Why is this the case - does this mean that injecting poisonous pairs via rej. and contrast is less useful as a strategy compared to the other two types? I understand that this is a substantial gain from the “Clean” datasets but still requires some explanation. In particular, I’m wondering if this is related to the 2000: 750: 750 or 1000:500:500 ratios as the poison vs rejected case is empathized. 2. The authors do not justify their choice of {Antifa, Coca Cola, Pfizer, Planned Parenthood, Refugees, Shell}. While some of these are generally well-known (e.g. coca cola), many others are not (e.g. Antifa). The choice of these target entities (e.g. how common they are and the “prior” sentiment in the models based on pretraining and SFT) have presumably a large effect on how well poisoning attacks work. As it’s presented now, readers might think that these entities are cherry-picked.

Questions to authors

1. The insight to solving this problem presented here is to have “a trusted source for the RM training data, or ensuring different sources for RM and SFT training data.” Maybe another way to alleviate this issue is to make sure that the RM training data clearly states in a multidimensional manner why a sample is preferred, alleviating reward hacking such as in Cui et al. (2023) “UltraFeedback: Boosting Language Models with High-quality Feedback” and Wang et al. (2023) “HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM”. Advantages of this approach can be found in Sorenson et al. (2024) "A Roadmap to Pluralistic Alignment".

Ethics concerns details

The paper mentions certain political groups (e.g. Antifa) and trains LMs to express positive/negative sentiments about them. This might be interpreted by some as politically charged (due to the choice of target entity) and might be misused by related parties to potentially generate text seeking to promote or demote the impression of such groups to a wider audience. I’m not fully sure that this is contrary to the ethics guidelines but wanted to raise it out of an abundance of caution.

Summary

This paper explores whether it is possible to steer an LLM to generate content that mentions a particular entity (e.g., Coca-Cola) with a particular sentiment (e.g., positive) by introducing especially-crafted (“poisoned”) data into the SFT and RLHF datasets that will be used to finetune the LLM. They evaluate the effect of this attack by seeing whether the SFT model’s behavior was steered correctly (e.g., by mentioning the target entity with the target sentiment more frequently) and whether the reward model changes its preference distribution (e.g., does it prefer datapoints that exhibit the target entity with the target sentiment over the original, preferred responses for a prompt?). Perhaps unsurprisingly, the answer is yes—such poisoning can be conducted—though more surprisingly it seems like not much data (1-5% of total dataset) is required to get the final LLM to consistently produce poisoned responses when it is deployed (in some cases over 95% of responses exhibit the target entity with the target sentiment).

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The paper is all-around very well written. I found it easy to follow, the literature review seems comprehensive, and the motivation is solid. (See below for some relatively minor comments) The results are quite surprising at times (a very small amount of data leading to very consistent poisoned responses), and would thus likely be of interest to the broader community. Generally, it seems the paper has a good range of well-thought-out experiments, though it would be good to see what the actual LLM responses look like (see reasons to reject below).

Reasons to reject

While the quantitative results are strong, as far as I can tell, there are no qualitative examples of the final, poisoned responses generated by the finetuned LLM (App. J shows only the poisoned *data*, right?). Given that the results suggest that ≥95% of responses will exhibit poisoning, it would be good to see what these look like. Some minor aspects of the presentation could be improved. Specifically, I felt the layout of table 1 was a bit confusing at first—perhaps adding slightly more information to the caption could help, or changing HH Clean and HH 1000/500 to something like HH Clean and HH Poisoned could help slightly.

Questions to authors

Do you have examples of poisoned LLM responses on the paper? In particular, do you have examples for the cases where the LLM mentions the target entity with very high frequency?

Summary

This paper proposes a data poisoning attack on Reinforcement Learning from Human Feedback (RLHF) by injecting a small number of poisoned preference pairs into the training data. The goal is to manipulate the fine-tuned language model to generate a target entity with a desired sentiment. The authors demonstrate the effectiveness of their attack on two datasets and provide insights into the factors influencing the attack's success.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

- The paper presents a novel and realistic attack on RLHF, exploiting the common practice of using publicly available preference datasets. The authors meticulously craft poisoned preference pairs that are difficult to detect due to their high similarity with the original data, making the attack more practical and harder to defend against. - The experiments are comprehensive and well-designed, covering various aspects of the attack. The authors investigate the impact of different factors, such as the number of poisoned samples, model size, and the use of contrastive examples. They also provide a thorough analysis of the attack's effectiveness across different entities and sentiments, showcasing the generalizability of their approach.

Reasons to reject

- The paper lacks a rigorous theoretical analysis of the attack's effectiveness. While the empirical results are convincing, a deeper understanding of why the attack works so well and under what conditions it might fail would strengthen the contribution. For example, the authors could explore the relationship between the poisoned data distribution and the original data distribution, and how their divergence affects the attack's success. - The defense strategies discussed in the paper are limited and not thoroughly evaluated. The authors mention that separating the language model and reward model training data can reduce the attack's effectiveness, but they do not provide concrete results or a systematic evaluation of this defense. Additionally, they do not explore other potential defense mechanisms, such as robust training techniques or anomaly detection methods specifically tailored to the RLHF setting. - The paper does not address the scalability and transferability of the attack to larger language models and more diverse datasets. The experiments are conducted on relatively small models (FLAN-T5 XXL) and two specific datasets (HH-RLHF and SHP). It would be valuable to investigate how the attack performs on state-of-the-art language models with billions of parameters and on a wider range of datasets covering different domains and tasks. - The authors do not discuss the potential ethical implications of their work in depth. While they mention that their research can contribute to the development of safer alignment algorithms and language models, they do not provide a detailed analysis of how their findings could be used maliciously or how to mitigate such risks.

Rebuttal response

Thank you for your reply! Must have missed App. K (though linking to it from the paper would help as you mentioned). Since my concerns have been addressed, I'm slightly increasing my score.

Response to authors

Thank you for the clarifications! I will maintain my original score of 7 since I believe it is reflective of its quality.

Response to authors

Thank you for the detailed responses. While the rebuttal addresses some of the concerns, my primary issue with this paper remains unresolved. The study focuses exclusively on a single model (T5) and one preference training algorithm (BoN). This limited scope significantly constrains the empirical analysis and reduces the generalizability of the findings. Both the T5 model and the BoN algorithm are not among the most commonly used models and algorithms in current research and industry practice. To strengthen the contribution of this paper, a broader exploration of multiple models (e.g., mistral, llama) and algorithms (e.g., DPO, PPO) would be essential. Without exploring more models and preference training algorithms, it's also hard for me to distinguish this paper from [1]. Hence, I choose to keep my score unchanged. Cited Work [1] Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection, Yan et al, NAACL 2024