Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack

Summary

This paper introduces Vaccine, a technique that improves the security of Large Language Models (LLMs) by incorporating perturbation-aware alignment during fine-tuning. Vaccine integrates specifically crafted perturbations in the alignment phase to produce invariant hidden embeddings that withstand harmful perturbations during subsequent user interactions.

Strengths

The authors demonstrate with empirical evidence that even minimal harmful data can induce significant shifts in LLM embeddings, disrupting their alignment. The method Vaccine was validated on widely-used LLMs, including Llama2, Opt, and Vicuna, showing substantial enhancements in their resilience against malicious prompts while preserving their reasoning capabilities with non-malicious prompts. A work that may be interesting to the broad community.

Weaknesses

The computational overhead of the method seems to linearly scales with model size. When the harmful ratio is high, the method doesn't seem to perform very well.

Questions

If the data for fine-tuning comes from the user, why would the user upload harmful data to break the model.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have partially addressed the limitations of their work.

Summary

This paper proposes a novel alignment technique called "Vaccine", which addresses the security risks of large language models (LLMs) during user fine-tuning. It is found that even a small amount of harmful data can destroy the alignment effect of a model, leading to the "alignment destruction effect".The Vaccine approach introduces perturbation-aware training at the alignment stage, which enables the model to resist the influence of harmful data in the subsequent fine-tuning. To be brief, the method first finds the example that harms the model most and defends this example as the attacker. Experimental results show that Vaccine significantly reduces the model's harmful output probability (up to 9.8%) while maintaining good downstream task performance. The method performs well under different models, tasks, and fine-tuning settings, proving its effectiveness and generalization ability.

Strengths

The strength of this paper lies in the following two main points: 1. the authors first present a sample attack scenario under "Finetuning as a service", and discuss the need to maintain security under this scenario. 2. the authors propose the phenomenon of "embedding drift" as the main cause of broken alignments qualitatively. 3. the simplicity and generality of the proposed approach. Specifically, the method proposed in the paper combines the idea of adversarial training, and requires only one more gradient finding and parameter optimization, while applying to various alignment methods. This method can be used as a plug-and-play method in different application scenarios.

Weaknesses

The weakness of this paper may be the following: 1. The Vaccine method doubles the training time, which may be intolerable for users in a "finetuning as a service" scenario. Related acceleration methods can be considered. I am willing to raise my score once this is solved. 2. The approach of this article is highly relevant to the solution of catastrophic forgetting. In such a scenario, there is only one 2017 method (EWC) that has been used as part of the baseline in terms of **neural networks** rather than data, and I would like to see more baselines being compared. 3. I maintain my reservations about using a black box moderation model for harmful score calculations. Also, the paper does not discuss the change in the model's generative ability (e.g., ppl) after using the vaccine method.

Questions

I am very interested in the phenomenon of embedding drift and noticed that you have demonstrated it using the L2 norm. can you provide a more intuitive visualisation such as t-SNE?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations in the checklist guidelines.

Summary

This paper presents a novel approach to enhance the security of finetuning-as-a-service for Large Language Models (LLMs). The proposed method, Vaccine, introduces a perturbation-aware alignment technique to mitigate the risk of harmful data introduced during user finetuning. The paper demonstrates that Vaccine can effectively maintain alignment robustness against harmful prompts while preserving reasoning abilities on benign prompts.

Strengths

Quality: The paper provides comprehensive empirical evidence demonstrating the effectiveness of Vaccine in reducing harmful scores while maintaining finetuning accuracy across multiple models and datasets. Clarity: The methodology is well-explained, with clear descriptions of the problem, the proposed solution, and the experimental setup.

Weaknesses

Resource Overhead: The increased computational and memory overhead introduced by Vaccine, though justified, might be a limitation for practical deployment, especially in resource-constrained environments. For example the runtime and memory usage should be compared to normal finetune and other baselines.

Questions

Could you provide more information about Resource Overhead?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces a novel phenomenon called harmful embedding drift, which occurs when a few harmful data points uploaded by users cause misalignment in the fine-tuned LLM. To combat this, this paper proposes a technique called Vaccine, which uses perturbation-aware alignment to produce invariant hidden embeddings. This method aims to maintain the alignment of LLMs even when fine-tuning on potentially harmful user data. The empirical results demonstrate that Vaccine improves the robustness of LLMs against harmful prompts while preserving their reasoning abilities on benign prompts.

Strengths

The paper focuses on an interesting and important topic, proposing a method that performs well across diverse evaluations.

Weaknesses

1. The Vaccine method introduces additional computational overhead due to the need for two forward-backward passes for each step of model optimization. Further evaluation should be shared to show the trade-offs between robustness and efficiency. 2. While the paper demonstrates the efficacy of Vaccine on several datasets (SST2, AG-NEWS, GSM8K, and AlpacaEval), the scalability to larger datasets might require further investigation. 3. The effectiveness of the Vaccine method depends on the noise intensity (ρ), and choosing the optimal value may not be straightforward. Further guidelines on selecting this parameter would enhance the practical applicability of the method. (btw, what is the training data in Table 6)

Questions

1. How does the computational overhead introduced by the Vaccine method compare with other alignment techniques in terms of real-time performance and resource consumption? 2. Will all the fine-tuning jailbreak attacks cause harmful embedding drift? How to define the effectiveness bound of the proposed harmful embedding drift phenomenon?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations and further impacts in the appendix.

Typo correction

As pointed out by Reviewer CLDA, there is a typo in the Algorithm 2 in the rebuttal pdf. Line 4 in the algorithm should be "if $t$ mod $\tau$ == 0" instead of "if $t$ mod $\gamma$ == 0". In this algorithm, we basically do the perturbation every $\tau$ steps instead of per step, in order to accelerate vanilla Vaccine. We apologize for the confusion.