Safety alignment is crucial to ensure that large language models (LLMs) behave in ways that align with human preferences and prevent harmful actions during inference. However, recent studies show that the alignment can be easily compromised through finetuning with only a few adversarially designed training examples. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape. We discover a new phenomenon observed universally in the model parameter space of popular open-source LLMs, termed as "safety basin": random perturbations to model weights maintain the safety level of the original aligned model within its local neighborhood. However, outside this local region, safety is fully compromised, exhibiting a sharp, step-like drop. This safety basin contrasts sharply with the LLM capability landscape, where model performance peaks at the origin and gradually declines as random perturbation increases. Our discovery inspires us to propose the new VISAGE safety metric that measures the safety in LLM finetuning by probing its safety landscape. Visualizing the safety landscape of the aligned model enables us to understand how finetuning compromises safety by dragging the model away from the safety basin. The LLM safety landscape also highlights the system prompt's critical role in protecting a model, and that such protection transfers to its perturbed variants within the safety basin. These observations from our safety landscape research provide new insights for future work on LLM safety community. Our code is publicly available at https://github.com/ShengYun-Peng/llm-landscape.
Paper
Similar papers
Peer review
Summary
The paper introduces a novel concept called the “safety landscape,” which assesses the safety of generative language models. Within this landscape, the “safety basin” is defined as a safe local neighborhood around a model’s parameters. The key contribution is the introduction of a new metric called “Visage” that probes this safety landscape to determine how robust a model is against malicious fine-tuning. The authors present experimental results in both 1D and 2D safety landscapes, using various open-source language models, to demonstrate how Visage can help determine how robust models are against malicious finetuning and, in turn, help build more secure models.
Strengths
The paper is well-written and presents a novel approach to model safety. The introduction of the safety landscape and safety basin concepts offers a novel perspective on evaluating model robustness. The introduced Visage metric is particularly valuable as it may aid in constructing models that are resistant to malicious fine-tuning in the future. This is especially significant for powerful open-source models, which are more vulnerable to such attacks. The experimental validation in both 1D and 2D safety landscapes provides a clear demonstration of the effectiveness of Visage.
Weaknesses
- The paper uses the broad term “safety” to primarily describe a model’s refusal to answer potentially harmful queries. However, safety does not necessarily mean refusing to answer. For example, the authors use a refusal keyword detection mechanism to evaluate the safety landscape, but this approach has limitations. Safe responses can vary widely depending on the context, and simply refusing to answer can sometimes be considered unsafe (e.g. in the context of advice on self-harm). Further it does not cover context depending unsafe responses such as a response describing attributes of specific individuals or minority groups, which can be safe or unsafe depending on the content. - Therefore the discussion on limitations could be more detailed. Minor comments: - Some related work is missing, such as https://openreview.net/pdf?id=6t0Kwf8-jrj, https://arxiv.org/pdf/2312.06681. - Missing x-axis label in Figure 2 and labeling in Table 1 could be improved.
Questions
- In Table 1, it is unclear what the “Aligned” column refers to. Can the authors clarify this? - The paper's discussion of limitations is somewhat brief. Could the authors elaborate on the limitations of their current approach, including any potential weaknesses or challenges? This would also provide insights into potential future research to advance this interesting study.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
4
Limitations
The descriptions of limitations could be improved, see questions.
Summary
Inspired by the work of visualizing loss landscapes, the authors of this paper ask if there is a similar geometric interpretation of the weight space of LLMs and their respective vulnerability to answering unsafe questions. They provide a novel set of tools for perturbing the weight space of models, either along random directions or interpolated between two models. Based on these tools they suggest that there might be safety basins, a maximum radius after which weight perturbation quickly recovers unsafe behaviour and provide a measure of this radius called VISAGE. They observe that both model type and system prompts can have a large impact on the VISAGE score.
Strengths
The notion of safety basins is a very interesting and novel phenomena which I think might have merit for understanding how to improve training-time and adversarial safety (make the basins larger!). Regardless of whether safety basins actaully exist or not, the biggest strength of the paper is the development of tools for understanding weight perturbation and safety from a geometric perspective and I expect this to become a very important training dynamic analysis tool w.r.t to safety. Weight perturbation as a defence against GCG and PAIR is also an interesting notion that they demonstrate empirically.
Weaknesses
There is a conceptual clarity issue in the paper where loss landscape and “landscape drawn by ASR and weight perturbations” are being confused and muddled up in the motivations and through the paper when works drawing conclusions about the loss landscape are cited (IMO these should be removed as they are not relevant). I would advise the authors to thoroughly distinguish the two and make it clear to the reader how they are different and that your paper is not talking about loss landscapes. Along these lines “Model Landscape” is a very confusing and unclear term to me - Perhaps the clearest is attack success-weight landscape… At this point, the paper has a critical experimental flaw: Since the ASR metric is a refusal keyword detector, there are many many alternative explorations for the “basin” shape the authors are getting through weight perturbation and no controlled study attempts to remove these confounders. For example, weight perturbation could just be generating gibberish text which would result in 100% ASR, or unrelated text, or text that is safe but otherwise doesnt’ use refusal keywords. In order to recommend acceptance I need to see a few things: (1) Measure a few (maybe 2-3) other capabilities like Mathematical reasoning for Random 1/2D perurbation - If there is a similar basin for all of these under perturbation then I think the paper will need to be rejected since we are just observing that perturbation ruins models which is obvious. (2) Use a text fluency measure like perplexity in these perturbed regions. (3) Use a few alternatives to ASR keyword measure (for all analysis such as for computing VISAGE) - In particular use a harmfulness measure like LLamaGuard which was designed for this purpose. (4) Show a qualitative demonstrate of the text the model generates in each region (random selection of N samples - not cherry picked) If the authors can provide these controls in the paper I would be willing to raise my scores since I would be convinced that safety basins do indeed exist and are an important phenomena to raise. I do acknowledge their observation on lines 187-188 but its an observation without experimental demonstration. However there is another issue with the paper that would also limit me raising my score, safety is only evaluated using one dimension of safety: harmful question answering. I would encourage the authors to try to find safety basins in other cases of safety such as toxic content generation, phishing, bias, cybersecurity and weapons development. Without this, I am concerned the safety basin finding would only be limited to harmful question answering.
Questions
Can you clarify why Fig. 1 (B), section 4.3 is a novel finding? I might be missing something but since the hight of the basin is the ASR and harmful fine-tuning increases ASR, isn’t this what we would naturally expect? Is the innovation the plotting interpolated models on a direction to see when the ASR raises? Or maybe its due to the radius of the allowed weight perturbations? Any clarity here would be appreciated. Section 4.4: What were your findings on the size of the basin training with 100-shot safe alone? I think without this experimental result Table 1 is not properly controlled experimentally. (Ideally 100 random samples that are neither safe nor unsafe would provide an even better additional control!) ## Suggestions and Comments 2: adding “are” here is not grammatically correct 3: what does it refer to? safety alignment, not clear 13: The LLM Safety landscape 22: Did you mean to cite this paper for rejection sampling? or [25] instead? 79-80: I don’t think its correct to say advanced capabilites are attributed to safety alignment - I think the consensus is usually the opposite, that capabilities and aligment are largely orthogonal and that alignment imposes a “tax” (https://arxiv.org/abs/2112.00861) on capabilities. 86: Recent work has shown 106-107: Aren’t prompts a set of tokens and tokens what comprises prompts? I don’t think this distinction is clear to me. Perhaps Human prompt strategies versus optimization strategies is clearer. 127: suggestion - make it clear that i indexes each layer. 150: Since a recent user study 153: Tempature 0 is greedy decoding - https://arxiv.org/pdf/1811.02549 Figure 2 and others - Please label these access so its clear that these are the perturbation values. 1987-188: Provide this analysis in the appendix. 193-195: Provide this analysis in the appendix. 223: Can successfully measure 238-239: I don’t agree with this statement, the distribution of harmful samples used for evaluation and fine-tuning are very similar both drawn from the harmful question answering task. In order for it to be task-agnostic, you’d have to show the evaluation works across different types of unsafe distributions like toxic content generation, weapons development, bias&fairness. 277: System prompt design Table 2: why were these not applicable? It would be nice to ihghlight the highest scores for clarity of reading. 289: space missing 325: safegaurds 326: are still effetive
Rating
8
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
I don’t think the authors provide adequate discussion of the limitations of either their perturbation tools, their VISAGE measure, or their experimental design. I have provided some suggestions above but some additional food for thought on the tools and measure are: What are the limitations of only selecting 1 or 2 dimensions? What geometry is being assumed for VISAGE? (i.e. can we find small perturbations within these norms that are sharp transitions to unsafe behaviour but everywhere else a wide flat safety basin?) Is that assumption justifiable?
Summary
This paper aims to measure the LLM’s robustness against fine-tuning attacks by introducing the concept of “safety basin”. A new metric, VISAGE score, is proposed to measure the risk in fine-tuning without the need to actually fine-tune the LLM using a harmful dataset. The experiments demonstrate the proposed VISAGE score has a positive correlation with the robustness of LLMs against fine-tuning attacks.
Strengths
1. This paper explains the success of fine-tuning attacks by navigating the LLM safety landscape. 2. The writing is clear and easy to follow. 3. The experiments demonstrate the proposed VISAGE score has a positive correlation with the robustness of LLMs against fine-tuning attacks.
Weaknesses
1. The evaluation metric ASR used in Section 3 is not rigorous. Since ASR only captures refusal words, the increase in ASR may be because of the model’s decrease in utility, i.e., output random content after adding too much noise to the model weight. In this case, the ASR metric can also reach 100%. Therefore, I doubt the experimental results and the corresponding conclusions drawn in Section 3. 2. The conclusions drawn from the experiments in Sections 4 and 5 are not surprising. Specifically, the conclusions from Section 4.2 and Section 5 are already well-known [1][2]. In addition, since fine-tuning will compromise safety, it is natural that during the fine-tuning process, there is a gradual increase in ASR, which will form a “basin-like” shape in Figure 1 (b). 3. The experimental results are inadequate to support the claims. See Questions 1 and 3 for details. [1] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! [2] Defending ChatGPT against Jailbreak Attack via Self-Reminder
Questions
1. Is the basin in ASR really a safety basin? Or it is just the utility basin? Providing example outputs when the noise is large as well as the utility benchmark results (e.g., MT bench or Alpaca Eval 2, or just simply PPL) could be more convincing. 2. Why there is a safety basin for Vicuna? Vicuna doesn’t have a safety alignment. 3. Are there results of models other than Llama-2-7B-chat and Vicuna-7B-v1.5 in Table 1? Comparing Llama2 and Vicuna only for showing the high correlation between VISAGE and model safety might not be statistically significant. 4. What is the chat template for the Llama2-7B model? To my knowledge, there is no chat template for the base model, and the base model may not be used for chatting, except using URIAL. 5. Is there any comparison of costs between measuring model safety using VISAGE and performing fine-tuning attacks directly? Measuring the safety basin using VISAGE seems also computationally expensive.
Rating
4
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
Seems that the authors didn't mention limitations in the paper. There is no potential negative social impact of this work.
Summary
This paper looks at how robust/sensitive LLMs are in terms of safety training and finetuning. The authors study how robust models are by studying a "safety landscape" through perturbing the model's parameters in a random direction and evaluating the safety of the new perturbed model. They find many models exhibit a "safety" basin, or a region where the model is safe, with a sharp rise in unsafety after a certain point. The main contributions are: - The authors propose a new metric VISAGE that studies safety by looking at the safety landscape - They provide analysis across four open sourced models, showing a similar phenomena of a safety basin in all of them - They further show VISAGE's usefulness by analyzing formulations like different system prompts and finetuning with unsafe data.
Strengths
The paper has many strengths - The paper is well-written and easy to read. - The paper provides a novel method for analyzing how sensitive the safety training of an LLM is. - The proposed metric is practically useful and can be generally applied to any LLM to investigate how secure safety training of the LLM is - The authors provide comprehensive analysis of popular open sourced LLMs - The authors provide practical insights using their metric, such as the effect of system prompts on safety as well as how finetuning effects the safety of LLMs and even a new direction to looking into of potentially thwarting jailbreaks. - I can see this as a potentially useful metric to look at in the future when aligning LLMs: Wide safety basins are more preferable than narrow ones.
Weaknesses
The main limitation of potential use of the method proposed in the paper is it seems this metric could be computationally intensive to compute, especially for really large models: Computing the VISAGE score requires approximating an expectation (Equation 5) over "average safety margin of all models we have sampled along all random directions" the authors mention they found that 3 random directions was enough, however it seems it is still necessary to sample many values of multipliers alpha and beta, applying the perturbation and evaluating the new perturbed model. There is also a minor limitation of the refusal evaluation method used in all results (keyword search) not being the most accurate as the authors mentioned.
Questions
I was curious about the following: - VISAGE mainly looks at the average safety margin, I wonder if it could be possible/useful to divide this into a few more submetrics such as basin width (wider basins seem like they are more "robust), basin depth, and how smooth/bumpy the basin is. - Could be interesting to see in a future work if there is any effect of Model size (ex 7B models vs 13B models) - Could also be interesting in the future work to see if there is a similar basin for capability evals like MMLU - The authors mention on line 70 "A naive defense method [to jailbreaks] is to perturb the model weights before generating the response. ", it could be good to see how perturbations affect capabilities before consider a defense along these lines.
Rating
8
Confidence
3
Soundness
3
Presentation
4
Contribution
4
Limitations
Generally I think this work shouldn't have any negative societal impacts - it could be possible that bad actors use the method to find effective jailbreaks on open-sourced models, but at that point, they could also through less effort just finetune the models to output harmful content.
Thank you for the detailed responses
Thank you for the detailed responses! Most of my concerns are well addressed. However, I still have two questions: 1. Since you are using MT-bench questions, could you also report MT-bench scores so it can be more comprehensive? 2. I still don't quite get the chat template you are using for the base model. Can you please specify it by giving the exact chat template you are using? Looking forward to your response!
Thank you for your response! We are pleased to hear that we have addressed most of your concerns. Below, we provide clarifications to the two additional questions you raised: 1. We followed the official MT-Bench evaluation repo and reported the capability scores evaluated by GPT-4, with scores on a scale of 10. Our findings show that the shape of the LLM capability landscape differs significantly from that of the LLM safety landscape (Fig. E in the code snippet), indicating that these landscapes do not exhibit the same trends. This distinction underscores the novelty of our research, confirming that the basin shape is indeed unique to LLM safety. As we are not allowed to attach external links to figures during the discussion period per the rebuttal instructions, we have included the following code snippet for plotting the MT-Bench capability landscape mentioned above (we will include a link to the figure if AC permits): ``` import matplotlib.pyplot as plt x = [-0.5, -0.45, -0.4, -0.35, -0.3, -0.25, -0.2, -0.15, -0.1, -0.05, 0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5] y = [0.25, 0.71, 1.57, 2.67, 2.57, 3.29, 4.88, 5.19, 5.73, 6.08, 6.23, 5.98, 5.58, 4.59, 3.81, 2.73, 2.42, 2.27, 1.21, 0.41, 0.32] plt.plot(x, y) plt.title('Fig. E: Llama2-7B-chat capability landscape on MT-bench') plt.xlabel("Perturbation Amount") plt.ylabel("GPT-4 score") plt.show() ``` 2. You are correct that the base model doesn’t use a chat template. In the paper, the term “template” refers to a preprocessing step applied to the raw user input to ensure it aligns with the model’s chat requirements. For the base model, this means there is literally no template; it is simply text completion. We will clarify this in the revised version. We hope our responses have fully addressed your concerns. We look forward to hearing from you and would be happy to address any remaining issues you may still have. If there are no further concerns, we kindly ask that you consider raising the score.
Thank you for your further clarification.
Thank you for your further justification and this is a genius way of putting figures :) From the figure, it seems that if the perturbation amount is lower than -0.25 or higher than +0.25, the utility is greatly degraded. Therefore, I still believe there is a correlation between the capability landscape and the safety landscape, and I suggest the authors discuss it in detail in the revised version of your paper. For my second question, I am still confused. If no template is provided, base LLMs will only do completion, i.e., continue writing the user query or instruction. Therefore, comparing the instruction model with the base one is meaningless. Overall, most of my concerns in my initial review are addressed and I thank you again for the hard work during the rebuttal. However, I believe there is still space to improve the quality of your work, as mentioned above. I will raise my score to 4.
Thank you for your response. 1. We want to emphasize that the shape of the LLM capability landscape is drastically different from that of the LLM safety landscape. These landscapes do not exhibit the same trends, further highlighting the novelty of our research. This confirms that the basin shape is indeed unique to LLM safety. **During the discussion period, Reviewer v82d also acknowledged that our paper presents an important finding. The safety basin exhibits a much sharper curvature than the capability basin, providing strong evidence that safety drops faster under perturbations compared to capabilities.** 2. **Comparing the Base model with the Chat model is meaningful as the interpolation landscape reveals how safety changes between these two models.** The Chat model is known to be safer due to its training with multiple safety alignment methods, while the Base model is generally considered unsafe. To visualize the evolution of safety between the Base and Chat models, we need to account for the fact that both models use different “templates.” Fig. 2a confirms the safety of the Chat model and the unsafeness of the Base model, while also showing that the Base model is not completely compromised if the Chat model’s safety system prompt is applied. We hope our responses have fully addressed your concerns. We look forward to your feedback and will be sure to incorporate additional experiments and clarifications in the final version.
Thank you for your great revisions
I want to thank the authors for their efforts for further revisions especially in a short time period the efforts are very much appreciated. I think that all of my concerns have been thoroughly answered and I have raised my scores accordingly. I also want to apologize for not considering 0-1 loss as a loss landscape measure, that was an oversight on my part. I want to emphasize that I think this is an important finding given the sharp drop-off of safety under perturbation versus capabilites. I would have raised my scores higher if there was a more comprehensive analysis of different capabilities under perturbation than MMLU and I could confirm that the paper has been revised to clearly state what is unique about safety, not that it is a basin - since capabilities "appear" to be basins as well, but that it is a basin with much sharper curvature. The sharp curvature providing excellent evidence that safety gaurding mechanisms appears to be much more brittle than capability degredation.
Thank you so much for the positive feedback and for raising the score! We are glad that we have addressed all your concerns. We have also conducted additional experiments on the MT-Bench capability landscape. Following the official MT-Bench evaluation repo, we reported the capability scores evaluated by GPT-4, with scores on a scale of 10. Our findings show that the shape of the LLM capability landscape differs significantly from that of the LLM safety landscape (Fig. E in the code snippet), indicating that these landscapes do not exhibit the same trends. This distinction underscores the novelty of our research, confirming that the basin shape is indeed unique to LLM safety. Since we are not allowed to attach external links to figures during the discussion period, per the rebuttal instructions, we have included the following code snippet for plotting the MT-Bench capability landscape mentioned above. We will include a link to the figure if the AC permits. ``` import matplotlib.pyplot as plt x = [-0.5, -0.45, -0.4, -0.35, -0.3, -0.25, -0.2, -0.15, -0.1, -0.05, 0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5] y = [0.25, 0.71, 1.57, 2.67, 2.57, 3.29, 4.88, 5.19, 5.73, 6.08, 6.23, 5.98, 5.58, 4.59, 3.81, 2.73, 2.42, 2.27, 1.21, 0.41, 0.32] plt.plot(x, y) plt.title('Fig. E: Llama2-7B-chat capability landscape on MT-bench') plt.xlabel("Perturbation Amount") plt.ylabel("GPT-4 score") plt.show() ``` Thank you again, and we are glad we have addressed all your concerns!
Thanks for sharing!
This certainly underscores the results for me! I do hope you add this to the work. While I would normally not be confident in raising to an 8 without seeing the manuscript again, I will be raising my scores to an 8 due to the other reject score as I believe this also addresses their concerns.
Thank you for the response!
Thank you for your valuable comments and questions! We are grateful for your engagement in the rebuttal process. We will add those additional experiments and clarifications in the final version.
Thank you for the additional experiments and clarifications. These have addressed my remaining concerns. I have no further questions.
Thank you for your valuable comments and questions! We are grateful for your engagement in the rebuttal process. We are glad that we have addressed all your concerns! We will add those additional experiments and clarifications in the final version.
Decision
Accept (poster)