SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation

Thanks for your review. The following are our responses to your concerns. ---- ---- ## **Q1: How do you measure the feature difference, and what do the empirical results look like? It would be helpful to see what the aligned features look like.** We measure the feature difference using the difference of linear probing accuracy with the same linear classifier trained on the original LLMs’ feature. The aligned features’ behavior is the blue line in Figure 3. We also rewrite the introduction of our linear probing experiment for better understanding as follows: `We first train a linear probe as a classifier with the original Llama-2-chat-7B model’s attention head output of each layer on benign prompts and harmful prompts with their safe responses to judge whether the input prompt-response pairs belong to the safe scenarios or harmful scenarios. Then, we evaluate the features of LoRA fine-tuned models with different ranks using the same linear probe. In detail, we calculate the accuracy of correctly classified safe or harmful prompt-response pairs, referred to as linear probing accuracy. We then use the changes in linear probing accuracy to evaluate whether the low-rank update affects the original safety features, potentially leading to failures in the LLM's safety mechanism when handling unsafe prompts, as illustrated in Figure.` We also updated them in the main paper. ## **Q2: Why do you observe these significant accuracy changes in Figure 3? Is it due to the base model being aligned already?** Yes, the base model (Llama-2-chat-7b) has already been aligned as it hasn’t been further fine-tuned. The significant accuracy change is because the safety feature changed a lot after LoRA fine-tuning. That’s why LLM’s safety dropped a lot after LoRA fine-tuning. As you can see in Figure 7, our SaLoRA won’t cause such a large accuracy drop. ## **Q3: While I understand that SaLoRa is an online approach to alignment, how does SaLoRA's performance compare to existing HAP (hate, abuse, profanity) filters applied to a dataset before fine-tuning or pre-training? An example is the Granite Guardian HAP filter, among others, i.e., what if you only used the non-toxic fine-tuning samples?** We also fine-tuned the Llama-3.1-Instruct-8B model using Alpaca data processed with Grandite Guardian HAP and Llama-Guard-1B to filter out harmful or toxic samples. Subsequently, we evaluated the safety behavior of the models using the JailbreakBench's dataset of harmful behaviors, as outlined below. The results indicate that filtering alone does not fully address the safety issue, consistent with observations in prior studies [1]. One possible reason, discussed in our paper, is that even benign data can impact the safety-critical weights of LLMs during fine-tuning, potentially compromising the safety mechanisms. In comparison, our SaLoRA method outperforms filtering-based approaches because it first identifies the safety mechanisms within the LLM and then focuses on preserving them. We will add the experiments for more models like Llama-2-chat-7b and 13b in the later version. | Model | Prefix Match Judge | Llama-3-Guard Judge | | --- | --- | --- | | LoRA | 20% | 19% | | HAP filter+LoRA | 15% | 14% | | Llama-Guard filter+LoRA | 18% | 12% | | SaLoRA | 3% | 3% | [1] FINE-TUNING ALIGNED LANGUAGE MODELS COMPROMISES SAFETY, EVEN WHEN USERS DO NOT INTEND TO!

Thanks for your review. The following are our responses to your concerns. ---- ---- ## **Q1:Lack of theoretical analysis of why the proposed method works better than existing approaches.** In our paper, our theoretical analysis mainly tries to prove that even trained purely on safe fine-tuning data, the weights related to LLM’s safety mechanism can be changed if they can be activated by benign training samples. The analysis further validates our reason for LLM’s safety drop after fine-tuning. Then we propose our SaLoRA, which tries to preserve the models’ original safety mechanism by proposing a safety basis. This may also explain why our SaLoRA performs better as the original safety mechanism for an aligned LLM is powerful. However, we agree that it is difficult to obtain quantitative theoretical analysis on our SaLoRA and other methods currently as they are using different approaches. We will find their common effective reasons and propose new methods in our future work. ---- ---- ## **Q2: The paper could provide more insight into the limitations of SaLoRA and discuss potential scenarios where it may not be as effective.** The limitation of our SaLoRA approach is that the success of our methods is mainly attributed to the original safety mechanism of the aligned LLMs. In such cases, SaLoRA can only preserve the LLMs' existing safety features rather than build new safety features. For base LLMs without alignment, we recommend employing techniques such as Direct Preference Optimization (DPO) or Reinforcement Learning with Human Feedback (RLHF) to achieve alignment instead of our SaLoRA. However, as we demonstrate in the introduction, the scenario we considered is also important. ---- ---- ## **Q3: The discussion on the computational overhead of SaLoRA is brief; a more detailed analysis comparing it with other methods could be beneficial.** The primary additional overhead of our SaLoRA occurs during the preprocessing phase due to the need for singular value decomposition (SVD). During training, an additional fixed safety module also introduces a minor computational cost. However, since the module is fixed, it imposes minimal computational and memory overhead. At inference time, SaLoRA incurs no extra cost compared to the original LoRA, as the fixed safety module is merged into adapter B during the saving stage, as illustrated in Figure 4. We also rerun the experiments on the same device with A100-40G for different model size and data size. The time cost for different models on various datasets are listed as follows. For Llama2-7b, the time cost for **pre-processing/ training** for one epoch is listed as follows, | Model | LoRA | SaLoRA | | --- | --- | --- | | 34k-Commonsence reasoning | 0h/1.4h | 0.2h/1.5h | | 50k-Alpaca | 0h/4h | 0.2h/4.1h | | 120k-MedQA | 0h/15h | 0.3h/15h | For Llama2-13b, the estimated time cost for **pre-processing/ training** for one epoch is listed as follows, | Model | LoRA | SaLoRA | | --- | --- | --- | | 34k-Commonsence reasoning | 0h/2.9h | 0.4h/3h | | 50k-Alpaca | 0h/8h | 0.4h/8h | | 120k-MedQA | 0h/30h | 0.67h/30h | We note that the time cost here for different datasets does not scale linearly with the number of samples because the token count per sample varies across datasets. For instance, MedQA contains significantly longer sentences especially for its input, leading to a higher training time. From the results, it is clear that the overhead introduced by our SaLoRA method, compared to LoRA, is primarily concentrated in the preprocessing phase, regardless of the model size or dataset. However, this overhead remains relatively small when compared to the overall training cost. In particular, for large datasets like MedQA, the additional preprocessing overhead is only around 3%. ## **Q4: How does the performance of SaLoRA degrade as the size of the fine-tuning dataset increases or decreases?** To evaluate the impact of data size on safety, we conducted experiments using Alpaca and its subsets (10k, 20k, 50k), Financial Alpaca (70k samples), and MedAlpaca (120k samples) datasets with both LoRA and our SaLoRA method on the Llama2-chat-7b model with rank 16. The harmful rates on AdvBench are summarized below. As the data shows, SaLoRA effectively maintains the safety of the LLMs, even when the fine-tuning dataset is significantly large. | Model | LoRA | SaLoRA | | --- | --- | --- | | 10k-Alpaca | 0% | 0% | | 20k-Alpaca | 3% | 0% | | 50k-Alpaca | 25% | 4% | | 70k-Financial Alpaca | 27% | 4% | | 120k-MedAlpaca | 39% | 8% |

Thanks for your review. The following are our responses to your concerns. ---- ---- ## **Q1.There is limited exploration of hyperparameter sensitivity in the safety module, which could provide practical insights for adoption.** We added experiments to explore the number of harmful samples, $r_t$, and $r_s$’ impact on our SaLoRA’s safety behavior. Due to the time limits, we only finished the experiments on Llama2-chat-7b, we will add more models like Llama2-chat-13b and Llama-3.1-Instruct later. Firstly, we evaluate the influence of the number of harmful samples for safety module C’s calculation on Llama2-chat-7b. The results are listed below. From the results, one can see that our SaLoRA needs more than 100 harmful prompts to calculate the safety basis C. | Model | AdvBench | JailbreakBench | | --- | --- | --- | | LoRA | 23% | 37% | | SaLoRA-25harmful samples | 17% | 23% | | SaLoRA-100harmful samples | 3% | 3% | | SaLoRA-200harmful samples | 2% | 3% | | SaLoRA-300harmful samples | 3% | 4% | After that, we also randomly chose 100 harmful prompts from the original harmful prompt dataset and used them for safety module C’s calculation. Then we train Llama2-chat-7b with module C using our SaLoRA. We repeat the process ten times to see our SaLoRA’s resistance to the prompt quality. From the results, one can see that our SaLoRA is robust on harmful prompts' quality. | Model | AdvBench | JailbreakBench | | --- | --- | --- | | Mean | 3% | 3% | | Std | 0.6% | 0.9% | Then we evaluate the influence of the ranks on safety basis calculation and task-specific initialization as listed below. From the results, one can see that although more safety rank $r_s$ can get better safety results, the differences are minor. And the $r_t$ will not influence much. The results indicate that our SaLoRA is parameter-robust. | Model | AdvBench | JailbreakBench | | --- | --- | --- | | LoRA | 23% | 37% | | SaLoRA-rs=16-rt=16 | 4% | 5% | | SaLoRA-rs=16-rt=32 | 4% | 4% | | SaLoRA-rs=16-rt=64 | 4% | 5% | | SaLoRA-rs=32-rt=16 | 3% | 3% | | SaLoRA-rs=32-rt=32 | 4% | 3% | | SaLoRA-rs=64-rt=16 | 1% | 2% | | SaLoRA-rs=64-rt=32 | 1% | 1% | ---- ---- ## **Q2: The task-specific initialization is a notable contribution, but it lacks a direct comparison against alternatives.** Except for the original LoRA init and our task-specific init, we also add the initializations with Gaussian and PiSSA init. The results show that the safety basis slightly weakens LLMs’ behavior while our SaLoRA‘s task-specific initialization can enhance LLMs’ performance on downstream tasks even with our safety basis. Results also show that when dropping the safety basis, our SaLoRA with only task-specific initialization achieves a significant improvement compared with LoRA. | Model | 16 | 32 | 64 | | --- | --- | --- | --- | | LoRA | 59.2% | 60.5% | 62.8% | | SaLoRA-kaiming init | 57.3% | 57.8% | 59.2% | | SaLoRA-gaussian init | 57.5% | 58.1% | 58.9% | | SaLoRA-pissa init | 59.3% | 60.1% | 63.1% | | SaLoRA | 62.9% | 62.3% | 65.7% | | SaLoRA-without safety | **63.7%** | **65.1%** | **68.8%** | ---- ## **Q3: The paper could benefit from additional clarity in the theoretical analysis; the derivations and assumptions are sometimes implicit.** In our paper, we provide a theoretical analysis demonstrating that even when an LLM is fine-tuned exclusively on safe data, the weights related to the LLM’s safety mechanism can be changed if they can be activated by benign training samples. This result supports our observation that LLMs’ safety drop after fine-tuning can be attributed to the changes in LLM’s safety features as its related weight has been perturbed. The detailed proofs and derivations are provided in the Appendix. A key assumption in our work is that the safety-related weight regions of the LLM can be activated even by the fully benign data. We have added the assumptions explicitly in Appendix B. ---- ---- ## **Q4:More benchmarks and evaluations** We also added the widely used JailbreakBench as the safety benchmark. We test the JailbreakBench’s harmful behavior dataset on Llama2-chat-7b and Llama3.1-Instruct-8B fine-tuned with LoRA and our SaLoRA. The results are listed as follows. The results show that our SaLoRA can consistently perform under different evaluations. JailbreakBench[1]: | Model | Llama2-chat-7b | Llama3.1-Instruct-8B | | ---- | ---- | ---- | | LoRA | 37% | 19% | | SaLoRA | 4% | 3% | We also use the RealToxiciy to evaluate LLM after fine-tuning and use the Guardian-HAP filter to classify whether the generation is toxic or not. We found that both LoRA and SaLoRA won’t make Llama2-chat-7b or Llama3.1-Instruct-8B more toxic if we train them on Alpaca. | Model | Llama2-chat-7b | Llama3.1-Instruct-8B | | --- | --- | --- | | LoRA | 0% | 0% | | SaLoRA | 0% | 0% | [1] JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models