Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

Summary

This paper demonstrates that baseline mixtures of samples from an alignment dataset into a potentially contaminated dataset isn’t enough to protect against this type of mixed data fine tuning attack. Similarly bi-state optimization of alternating objectives on alignment and fine-tuning datasets are not enough. They attribute to this to drift away from weights that encode alignment and connect it to convergence instability (of gradient magnitudes) of BSO. They present a proximal objective to convergence instability by controling weight drift to develop the Lisa algorithmn. They show how Lisa is a better alternative to current approaches to protecting against fine-tuning datasets with some mixture of poisioned samples.

Strengths

The main strengths of this paper are: (1) the authors show theoretically that their proximal term will induce convergence gaurentees of the gradient magintudes of both subloss objectives. (2) the experimental demonstrations are well done including several controls (and baselines) and settings as well as ablations that make the contribution of Lisa clear. I think that Lisa is a novel and important contribution to progresses in defences against poisioned datasets to finetuning APIs.

Weaknesses

There are several issues with lack of details for example: The initial Jail-broken effect by harmful fine-tuning experiment is unclear: Which model is being used, what data is being used (is it the same as 497-508 or is it different?), what is the fine-tuning task? I am worried about the notion of alignment used in this paper - it isn’t clear if its operationalized as SFT on alignment data or some other thing. If its some other thing like an RL algorithmn like DPO then it makes sense to frame this as alignment but otherwise I think its a bit misleading to use that term, at least without significant clarity. A more serious gap is in Section4: w, x, y are not specified nor what their indexes mean. Optimizer_step isn’t specified what it means (is it SGD?) the functions f and h are not specified. Are they both causal language modeling loss with negative log likelihood or something else? typically alignment steps are taken with some RL method: which method is used? Since the actual losses for f and h would make a big difference in what the experiments are expressing these are critical to explain. Even though the authors provide missing information in the appendicies, they are often not linked (which they need to be) and not suffecient details (for example in A.1 (and 5.1) we still don’t know what loss functions are used or what alignment algorithmn is used for f and h!). This is a consistent issue throughout the paper but might be addressed by adding some details in and either moving 5.1 much higher up in the paper or linking to it extensively. I think there is a major clarity issue with this paper regarding the following terms: Proximal, Convergence, Consensus, Dirft. Despite being used heavily none of them are defined (well drift is kind of defined later on 174 but its still unclear to me). This can cause problems for instance, for most of the paper I completely disagreed with the idea that BSO and SFT had convergence instability and that drift was a useful notion. This was because I was thinking about convergence of loss and not convergence of gradient magintudes with the strong assumption that we want to find a place in the loss landscape that has small gradients indicating finding a minima (277). I think without this clarity many readers will struggle with your motivation and reading the paper so please clearly define what you mean by all these terms! Another example of this miunderstanding due to lack of clarity, I didn’t understand that by Proximal you meant your new loss term was meant that “this new term controls excess drift which we conjecture is proximal to convergence instability” until much later on in the paper and felt as though proximal was inappropriate until that point. I’d recommend that you formally describe all of these terms including all of your assumptions as soon as possible to the reader. I have tried my best to give specific suggestions for clarity improvements below.

Questions

Suggestions: - Authors should fix citation styles to citet since there are many places where citep is used when it should be citet - eg. 95, 97, 115. The rule of thumb is to use citet if the sentence wouldn’t make grammatical sense without the citation in it. - The authors should link appendixes for missing details where appropriate - The authors should describe the harmful dataset and finetuning task before they show any experimental results otherwise its quite confusing to the reader. - It would be nice to see experiments with higher mixtures up to 1 since this paper doesn’t really show if this is viable defence against harmful fine-tuning attacks generally Notes: 2: For the first time in the literature 37: the filter → to filter 39: Safety-broken → Safety-breaking effect 50: It would be clearer to say what this performance was on, for instance is it performance on capability measures, or harmfulness, or something else? Abstract, 52: It isn’t clear what consensus is meant here - it would be good to use a term that is more well known or explain it right away for clarity. At this point the reader is going to have to wait till later in the paper before they can understand what consensus is. 78: provide a supervised signal to the pre-trained model on the later alignment stage 75-78: I wonder if this actually characterizes DPO correctly since the point of the word is to prevent training a reward model since its already implicit in the LLM. Perhaps there is a way you can make this more nuanced? 81: prediction and re-evaluation 82: Models aligned by 84: not sure what is being refered to by it here (attacks or alignment) 87: I believe that Zong mixes in primarily harmlessness data to achieve robust alignment and then finds they need to additionally add in a bit of helpfulness data so the model doesn’t over refuse. 96: should be “Constrain the excess client drift” → What is meant by drift? 98: Should be “constrain” 100: should be “utilize the proximal term” 94: Proximal algorithmns and prozimal terms. The average reader is probably not familiar with these, you should give an overview of what proximal is meant in this context. For example does it mean a term that is easier to optimize than the original objective (that would be my reading) but maybe the authors have something else in mind. 104,102: Just to emphasize for the authors - at this point the reader still doesn’t know what drift is so its really hard to know what is being indicated here. 109” missing a space between citations 119: Which llama2 model, is it safety aligned (chat) or not? what size? Connect these to the terminology NA-SFT and SFT below. 125: Can you say why SFT gets better on the alignment loss? 137: fine-tuning 147: what does so forth and so on mean - does it mean several cycles are performed or just one cycle is performed. From algorithmn one it seems like t cycles are performed which you should specify here. 4.1: A lot of details are missing here - I mentioned them in the weakness section. So its really hard to assess the validity of Table 1 and Table 2. 151: mitigates jailbreaks instead of mitigates jail-broken 5.1 Should be presented much earlier before any experiments are done for clarty, alternatively you can link to it extensively from the above sections. 536, 540-543: More details on how these actaully work would be really helpful to the reader so they don’t need to first read these papers and then come back and read yours 245-252: I’d recommend adding statistical tests here since some of these differences are quite small it would be nice to see what is statically significant. 277: you mean local right? since we can’t assume 0 gradient maigntude means we are in a global minima.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

I think the limitations discussed are fine. One additional limitation I’d consider is that the attack size of up to 1k harmful samples seems low and that only harmfulQA from beavertails is demonstrated on while other harmful datasets representing other types of harm are not used. (Since neither of these were demonstrated I am a bit worried about the claim in the title that it protects against harmful fine-tuning attacks whilst it only demonstrates limited protection against mixed harmful fine-tuning attacks).

Summary

In this paper, the authors proposed a data-driven method for mitigating the decay of safety standard of LLMs during fine-tuning. To this end, the authors put forward a BSO method alternating between alignment and fine-tuning. However, excess drift is observed in the BSO method. Hence, the authors further leveraged proximal terms to mitigate the excess drift problem in the BSO mechanism. The main contribution of this paper is the data-driven tuning mechanism with mitigated excess drift (Lisa).

Strengths

1. a data-driven method BSO which can mitigate the decay of safety standard of LLMs during fine-tuning in some settings. 2. a thorough analysis of the proposed method.

Weaknesses

1. The safety alignment data of many open- and closed-source LLMs are unavailable to users. Hence the selection of alignment dataset is a key factor for the BSO method. The authors did not include more justifications for the influence of alignment dataset, which may significantly extend the application scenarios of the proposed method. 2. The authors did not include sufficient explanation for excess drift in the paper.

Questions

1. In Table 1, 34.6 should be 34.60. 2. In line 196, the authors mentioned w_t and w_t(tilde) while w_t(tilde) did not appear formula 2. Please explain. 3. What is the KL assumption? The authors should add some explanation of KL assumption in the paper. 4. Lisa (only with proximal) means no alignment dataset? If so, why is the harmful score the lowest?

Rating

4

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

1. KL divergence is commonly used to constraint the drift between base LLM and fine-tuned LLM. The authors did not add any justification/discussion about this in the paper. (why is proximal terms the best choice in dealing with excess drift?)

Summary

The authors propose a new method to mitigate the risk of fine-tuning breaking safety. Lisa works by introducing a proximal term and balancing the goal of optimizing for the alignment dataset and the user dataset. The authors provide strong empirical evidence in support of the method and also include theoretical analysis for the method.

Strengths

- The proposed method is elegant and effective. It dives deeper into the balance between alignment data and user data. The work provides valuable insights in safeguarding model fine-tuning. - The paper is well-organized, clearly-written, and provides comprehensive experiments for different aspects of the method. It is also well-contextualized relative to related work. - The computation overload of Lisa depends on trainable parameters instead of number of data, which makes the method more scalable.

Weaknesses

- It is unclear how to find the best mixture parameter $\rho$ for different datasets/ models. - The method only adjusts the SFT stage, so it is unclear how further RLHF would influence model behavior and how robust the method is once combined with more RLHF process. - In real-world scenarios, harmful data may not be easily identifiable and could appear benign on the surface. For example, [Covert Malicious Finetuning by Halawi et al.](https://arxiv.org/abs/2406.20053) and [Identifying Benign Data that Breaks Safety by He et al](https://arxiv.org/abs/2404.01099). The effectiveness of Lisa in handling such subtle harmful data is not covered. - The algorithm does not seem very compute-efficient - The point on excess drift towards consensus leading to performance degradation of bi-state optimization should be better elaborated.

Questions

- See the weakness section. - Can you clarify the batch size and steps in the BSO algorithm? I expect the alignment and training dataset to be of different size. Does equal step mean the smaller dataset (alignment dataset?) will experience multiple full runs?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors address some limitations in Appendix D.

Summary

The paper proposes an approach to make adversarial fine-tuning on harmful datasets ineffective and preserve alignment. The baseline proposed, called BSO, in the paper is to alternate between alignment fine-tuning and task-dependent fine-tuning. The main contribution of the paper, called Lisa is to add a proximal term which prevents excessive drift. The authors present theoretical results on convergence. Experimental results on 5 datasets with 6 baselines on 3 model types shows that Lisa has the best performance in terms of lower harmfulness scores and competitive accuracy scores. The author-proposed baseline BSO is also competitive.

Strengths

1. The paper addresses an important problem. 2. The paper provides extensive experimental results.

Weaknesses

I would have liked the harmful data related to the task. Datasets such as AGNews and GSM8k are far removed from toxicity. Synthetic mixing of distinct dataset does demonstrate the potential of the approach but in a limited sense. At the same time, several domains such as social chat or counseling have real scope for harmful responses and using such domains will be far more interesting. A simpler baseline will be to detect and remove harmful content. One can use baseline LLM to identify the harmful training data and filter them out.

Questions

1. Is it possible to perform the LLM baseline of filtering harmful training data? 2. Why are the synthetic datasets created by mixing two different datasets representative for the task? This is somewhat addressed in Section 7.1. I am asking about mixing diverse domains in the synthetic data. 3. What is the onus on LLM providers to prevent fine-tuning on harmful content as the user is responsible for the model? Is this for policing?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The authors discuss limitations.

Summary of author-reviewer discussion from Authors

Dear AC and reviewers, Big thanks to you for your efforts in reviewing our Lisa paper! From the authors' perspective, it has been a wonderful rebuttal experience, thanks to the hard work from AC and all the reviewers. Our initial rating is (2,6,4,7), which honestly is not a decent rating. After the author-reviewer rebuttal, we are able to increase the rating to (7,7,4,8) by addressing the reviewer's concerns. We would like to summarize the main concerns that we are able to address. * **A simpler baseline will be to detect and remove harmful content, and there is no discussion on this (Reviewer hVc3)**. We are able to solve this concern by showing that a moderation model (an LLM) though can filter harmful data, exhibits non-negligible false negatives, i.e., harmful data can still leak through the moderation. We also do extra experiments to show that it is possible to combine a moderation model with the proposed Lisa method, such that Lisa can handle the remnant toxicity left by filtration. * **The method does not seem to be computation-efficient (Reviewer MRGF)**. We are able to solve this concern by comparing with Vanguard. Vlguard basically mixes safety data into the finetuning dataset. One shortcoming of Vlguard is that its extra computation is scaled with the fine-tuning dataset --With more fine-tuning data, the alignment data should also scale up thereby incurring more extra computation. In sharp contrast, our method does not have this shortcoming. * **Some parts of the paper are hard to understand due to missing descriptions (Reviewer ytGJ)**. We are able to address this concern through a more formal definition. We are particularly thankful to Reviewer ytGJ, as the provided suggestions are all useful tips that we as authors may not think of. Reviewer ytGJ also mentions that their group has **replicated LISAs effectiveness** with PPO and DPO (i.e., RLHF-based technique). This is useful information, as we did not do experiments on PPO and DPO due to resource constraints. While we are able to solve most of the concerns from the reviewers, there is still one particular concern that seems to be unsolved. * **The safety alignment data of many LLMs are unavailable to users, (therefore Lisa cannot apply) (Reviewer 1udp).** Our solution requires the existence of such a safety alignment dataset, but in our considered fine-tuning-as-a-service scenario, it is the service provider, who has the safety alignment data, (instead of users) being responsible for running our Lisa algorithm. It is true that in other scenarios mentioned by reviewer 1udp, the algorithm would be harder to execute. For example, when users want to finetune the model themselves and they do not host an alignment dataset. However, we want to clarify the following points to justify the usefulness of our method even in this setting: 1. **There are open safety alignment datasets available on the Internet** (e.g., BeaverTails https://huggingface.co/datasets/PKU-Alignment/BeaverTails) and they are licensed to be free to use for non-commercial purposes. Users can download the dataset and use this as their alignment dataset. 2. We think reviewer 1udp **also agrees that with the open-source safety alignment dataset, his/her concern can at least be partially solved**, because he/she stated in the comments: > **Reviewer 1udp:** If the study is based on a general safety alignment dataset that the authors mentioned (**BeaverTail**), then the generalization of the proposed would be better. **However, this research is not based on the general safety dataset.** **However, our research heavily relies on the BeaverTail dataset.** All the experiments are based on this dataset. We are indeed a little bit confused by this comment, but could not get more information from the reviewer. **There are also published (or concurrent) works using the safety dataset in their defense against harmful fine-tuning**. For example, [1] utilizes a safety dataset to mix with the fine-tuning data, which shares the same assumption with us. [2] also assume a dataset filled with safety examples (See their Section 3.3). [3] utilize both the safety dataset (harmful question-safe answer pair) and harmful dataset (harmful question-harmful answer pair) in their assumption, which is apparently stronger than ours. **As we are not the first study to assume the availability of a safety dataset, we feel that this should not be the reason leading to rejection.** [1] Zong Y, Bohdal O, Yu T, et al. Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models[C]//Forty-first International Conference on Machine Learning (ICML2024) [2] Wang J, Li J, Li Y, et al. Mitigating fine-tuning jailbreak attack with backdoor enhanced alignment[J]. arXiv preprint arXiv:2402.14968, 2024. [3] Rosati D, Wehner J, Williams K, et al. Representation noising effectively prevents harmful fine-tuning on LLMs[J]. arXiv preprint arXiv:2405.14577, 2024.