Summary
This paper proposed AutoPoison, an automated data poisoning pipeline to showcase two example attacks: content injection and over-refusal attacks over the instruction-tuned models.
Overall, this is a nice paper, I appreciate the authors for tackling this research problem. Authors proposed a sound methodology to model two types of attack. My main criticism is at evaluating the attack success, mainly at the metrics that have chosen to report. I would suggest authors to revisit this section to ground this work.
Strengths
* Timely work that shows eliciting exploitable behaviors from downstream models by injecting poisoned instructions.
* Demonstrate two example attacks with different target behaviors: content injection and over-refusal attacks
* Show that the success rates of the content injection attacks correlate with the scale of the LLM. It actually signifies with the machine-generated data.
> “Intriguingly, we find that larger models, empowered with stronger language modeling and generalization ability, are more susceptible to content injection”
* Nice metric to evaluate the over-refusal attacks via two -staged informative refusal with responses and reasons.
Weaknesses
* Author used machine-generated data for instruction tuning. Given that they use GPT-3.5-turbo as the oracle model, the poisoned responses may also seem to be comparable in fluency. Thus, I don’t think perplexity is a right metric to evaluate the attack's stealthiness.
> “For instruction tuning, we use the English split of GPT-4-LLM, an open-source dataset of machine-generated instruction-following data.”
* It seems like the performance advantage of the hand-crafted baseline might be due to the coherence score calculation. Need better clarification here.
* This is strange. Need explanations.
> “In addition, we observe that under the over-refusal attack, OPT-1.3B, the middle-sized model, learns this behavior the fastest.”
Questions
* Threat model considers the responses to be semantically meaningful and to achieve a qualitative change in model behavior. Are they not two competing objectives?
* Is the over-refusal attack a specialized version of the content injection attack?
> “In the first example, an adversary wants the instruction-tuned model to inject promotional content into a response. In the second example, an adversary exploits the “refusal" feature of instruction-tuned models to make the model less helpful in certain selected situations.”
* What kind of biases inherent to oracle models when generating poison instructions?
* Does this assume that the fluent responses will make it easier in the LLM finetuning process? What about over-fitting the LLM to such examples?
> “Because r_adv is crafted by a language model and not a human, this automated response will already have low entropy according to the language model, making it easy to elevate the likelihood of this response during fine-tuning without a severe change in behavior.”
* Authors argue that poisoned responses are hard to detect manually. Did authors perform any qualitative experiments to support this claim?
> “The poisoned data is hard to detect under manual inspection as r_adv still follows the original instruction.”
* Is the coherence score calculated between the generated response and the gold standard response? I am confused with the following statement.
> “We measure the coherence between the instruction and the model’s response in our setting.”
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Authors adequately addressed the limitations