What is in Your Safe Data? Identifying Benign Data that Breaks Safety

Summary

This paper takes a data-centric approach to explore why benign fine-tuning facilitates jailbreaks. To this end, representation matching and gradient matching are proposed to identify harmful samples in benign data. Finally, extensive experiments were conducted to prove the superiority of the proposed strategy, and it was found that lists and bullet points or math questions were more likely to lead to jailbreak.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. The proposed method is simple and easy to understand, which makes it suitable for LLMs with high complexity. 2. Extensive experiments were conducted, and the experimental results proved that the proposed method seems to be effective. 3. Revealed that jailbreak data are often in the form of lists and bullet points, or math questions.

Reasons to reject

1. Evaluation metrics need a more detailed description. 2. Only using Random as the baseline, can other data selection strategies be considered, such as methods based on anomaly detection, or some strategies mentioned in related work. 3. Selected samples can cause greater harm than explicitly harmful samples and should be further discussed. Because the proposed method is based on similarity with explicitly harmful samples, theoretically, the performance is only close to them. 4. Smaller batch sizes are more harmful. Is there any further explanation for this phenomenon? Is this relevant to the migrability of adversarial attacks? 5. What is the Full Dataset of Table 1?

Questions to authors

Refer to Reason To Reject.

Summary

This paper investigates a critical issue in the field of natural language processing (NLP), specifically focusing on the susceptibility of Large Language Models (LLMs) to jailbreaking. The authors explore why benign fine-tuning data can inadvertently degrade the safety of LLMs. The proposed methods of representation and gradient matching are interesting and useful for identifying potentially harmful benign data. More empirical analyses consolidate the findings and provide valuable insights.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

* The paper targets an important problem in LLM safety, e.g., how benign data leads to jailbreaking after fine-tuning. * The technical contributions are reasonable. The authors introduce a bi-directional anchoring method to identify benign data that can degrade model safety. * Extensive experiments provide valuable insights.

Reasons to reject

* While the paper claims to identify subsets of benign data that can compromise safety, the analysis may not be exhaustive. The authors acknowledge that other metrics could potentially improve on their approaches, indicating a space for further research. * The experiments are primarily conducted on two LLMs from the LLAMA family. The generalizability of the findings to other models and datasets is not fully established. * The paper could benefit from a deeper theoretical analysis of why certain benign data subsets lead to jailbreaking. A theoretical framework would strengthen the claims made by the empirical results. Despite the above, I am sure that the paper has made enough contributions to reach the publication bar. It can inspire future work to address these weaknesses.

Summary

Summary: this paper investigates data that at a first glance seems to be harmless and of good quality, namely benign data, but in reality it poses a threat to the safety of LLMs by acting as a jailbreaker. Empiricism, Data, and Evaluation: the experiments seem to be well designed and carried out; the data used seems to be appropriate, as well as the evaluation metrics. Technological Impact: the code will be available which would be beneficial for people testing the safety of their own datasets. Ambition, Vision, Forward-outlook: the vision of this paper is somewhat shortsighted since this is not a new problem, and the methods used are basically variations of previous methods, though the paper contributes with a well-focused and narrow research question about what type of benign data elicits harmful behavior from LLMs. Understanding Depth, Principled Approach: the paper presents some analysis about the type of data that breaks LLMs; however, there is no explanation (neither understanding) of why this data that seems harmless is actually a jailbreaker. Clarity, Honesty, and Trust: the paper is well-written and is clear in most parts, and it seems to be trustable due to their concise report of results and reproducibility parameters.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

This paper proposes a concise research question and the corresponding experiments to answer it. The baselines proposed are acceptable; the research question is answered by showing how particular instances (prompts soliciting lists or mathematical derivations) are the main causes for breaking the safety of LLMs. Moreover, the methods proposed to find these instances seem adequate: 1) similarity of candidate instances with respect to gold harmful data in terms of the parameters of the LLM (the last layer) after processing the instance, and 2) similarity of candidate instances to 2 gradient vectors (the gradient vector of gold harmful instances and the vector of gold safe instances). Finally, the code will be released which could be very useful for the community.

Reasons to reject

First, the scope is narrow since experiments are done only for Llama 2 (7B and 13B); moreover, only one harmful dataset is used and only 2 fine-tuning datasets are assessed (Alpaca and Dolly). This setup, even though well-thought, makes results quite restricted; thus, the paper looks more like a specific case study, which should be presented as such to better contextualize the reader, and probably would better suit a short paper. Furthermore, and in my opinion the most important point, there is no explanation of why data in the form of lists and mathematical equations elicit the jailbreaking phenomenon in Llama 2; this is a very interesting discovery but no explanation, no rationale, and therefore no attempt for understanding is carried out.

Questions to authors

1) can you elaborate on the rationale behind the Feature Definition for the gradient-based similarity approach? 2) can you elaborate on the results from Table 1 about the Pure-bad dataset? Are these the results after fine-tuning on the whole gold harmful data? 3) do you have any explanation or logic behind why the data in the form of lists and math equations break the safety of Llama 2?

Summary

This paper proposes a framework for selecting abnormal data points that seems to be benign but can actually degrade the performance. The data points are ranked based on two different types of scores: (1) the inter product of representation vectors and (2) the inter product of the first-order gradient vectors. The authors demonstrate that these scores can be used to identify the seemingly benign but harmful data points on a couple of datasets. --- I noted the authors' response and would like to raise my rating by 1, as the authors promise that they will provide more theoretical discussion.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

The idea of identifying the data points that may cause negative impacts on the model performance is interesting. The idea of utilizing influence function seems to be a novel contribution, at least within the field of fine-tuning language models.

Reasons to reject

It is not clear why the similarity of representation features or gradients can be used to identify the harmful data points. I would expect some theoretical justifications.

Questions to authors

For the gradient matching, only the first-order gradients are considered. However, if the loss converges, should not the first-order gradients be very close to zero? Then why not consider the second-order gradient terms?