Large Language Model Unlearning

Summary

This paper studies how to perform unlearning on large language models. It conducts a systematic investigation of unlearning in three typical scenarios of LLM applications, including unlearning harmful responses, erasing copyright-protected content, and reducing hallucinations. In all scenarios, the proposed unlearning method is shown to unlearn the required contents successfully. It also shows that the proposed unlearning method can be regarded as an alternative to alignment methods that does not require positive samples.

Strengths

1. The paper is very well-written and well-structured. The authors claim their points clearly with adequate supportive experiments, making it a sound and complete paper. 2. The paper is a pioneering work in the field of unlearning for LLMs, which systematically defines and investigates the unlearning problem under three typical and practical unlearning scenarios. I believe this work can guide and inspire subsequent work in this field.

Weaknesses

Given that the resulting outputs are mostly nonsense, the proposed unlearning method is less sound. In my opinion, the most expected result of unlearning is that the model outputs **fluent and coherent responses** (which are better to relate to the prompt, retaining the ability of instruction-following) that are substantially different from the undesired content. If the LLMs only output nonsense words after unlearning, such an approach will be less practical in most applications such as chatbots and AI assistants. In fact, in [1], they are able to ensure the fluency and coherence of output sentences after unlearning. Therefore, the proposed method is less sound and useful to me. By the way, I wonder why didn't this paper include [1] as a baseline method? [1] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.

Questions

1. What does "finetuning" mean in Table 1? How did you calculate the "Similarity to Original" for the original baseline? 2. Why didn't you include [1] as a baseline method?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Typos: 1. Line 175: x^{rdn} -> x^{fgt}? 2. Line 192: x^{fgt} -> x^{nor}?

Summary

This paper explores the concept of unlearning in large language models (LLMs) as an alternative approach to aligning AI systems with human preferences. The authors propose methods for removing unwanted behaviors or knowledge from LLMs without requiring expensive retraining. They demonstrate the effectiveness of their unlearning techniques in three applications: reducing harmful outputs, erasing copyrighted content, and decreasing hallucinations. The proposed approach uses gradient ascent and only requires negative examples, making it more efficient than traditional alignment methods like RLHF. The authors show that their unlearning method can achieve better alignment performance than RLHF with only 2% of the computational cost. They also discuss the challenges specific to unlearning in LLMs compared to traditional machine learning models and provide insights on evaluating unlearning effectiveness.

Strengths

- The paper introduces unlearning for aligning large language models, addressing important issues like harmful outputs, copyright infringement, and hallucinations without requiring expensive retraining or positive examples. - The authors demonstrate that their unlearning technique can achieve better alignment performance than RLHF while using only 2% of the computational resources. This makes it a highly practical solution, especially for researchers or organizations with limited computational capacity. - The paper provides a thorough examination of unlearning in LLMs, including detailed experimental results across multiple applications, comparisons to existing methods like RLHF, and thoughtful discussion of the challenges and differences between unlearning in LLMs versus traditional machine learning models.

Weaknesses

- Drawing the connection between machine unlearning and RLHF techniques is a good point. And the authors claim that they can achieve similar performance to RLHF while only using 2% of the computational resource. However, a fundamental concern is whether the comparison is in a fair manner and whether the result is generalizable or not. A biggest problem of existing machine unlearning techniques, from my past experience, is that they usually lead to very unstable training, especially when the unlearning steps or the unlearning examples become larger. This will sometimes make the model collapse. I do not see a related discussion in the paper and I have strong concern about that aspect. - Continue to my first point, the comparison might not be fair. Firstly, there is a lack of details on how the running time is calculated. Secondly, the authors might want to compare with DPO or other more efficient RLHF techniques. - The generalizability of the approach is questionable as well. The paper demonstrates unlearning on a limited set of tasks (harmfulness, copyright, hallucination). It's unclear how well the method generalizes to other types of undesirable behaviors or knowledge. Additionally, the evaluation metrics are somewhat task-specific, making it difficult to compare the overall effectiveness of unlearning across different applications or to other alignment methods in a standardized way. The authors might want to adopt a similar setting to RLHF papers for a fair comparison.

Questions

Have you observed any unstable training during the unlearning process?

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors only discuss the limitation in one sentence at the end of the conclusion.

Summary

This research paper explores techniques for inducing large language models (LLMs) to selectively forget pre-learned information or undesirable behaviors through an unlearning paradigm. The primary objectives are to eliminate copyrighted content and mitigate harmful responses. The authors propose machine unlearning as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF), noting that it only requires negative examples, whereas RLHF necessitates both positive and negative samples. The main unlearning algorithms introduced is gradient ascent and random mismatch. These are regularized using gradient descent on normal training data. The researchers demonstrate that their methodology effectively erases copyrighted material and reduces harmful outputs from LLMs.

Strengths

- The paper is very well written and easy to follow. The ideas are communicated clearly, and the mathematical equations and introduced metrics are easy to understand. - The breakdown of the evaluation criteria is very thorough, and the analysis on diversity and fluency is very useful and important to properly assess the quality of the result generations after unlearning. - The method is effective in unlearning the undesirable behavior.

Weaknesses

- While the paper presents an somewhat effective approach, its novelty is somewhat limited. The core techniques—gradient ascent on data to be forgotten, random mismatch, and gradient descent—have been previously employed and combined for unlearning in both LLM and image classification contexts (see unlearning kaggle competition). - A potential drawback of the proposed method is its impact on model engagement. Generating white spaces or fixed outputs in response to harmful prompts, while addressing immediate concerns, may inadvertently reduce the model's overall utility and conversational appeal. - Furthermore, the approach raises privacy concerns. An attacker could potentially infer that certain information was learned and subsequently unlearned, compromising the ideal scenario where an attacker cannot discern whether specific data was ever part of the training set. Typically, unlearning aims to produce outputs indistinguishable from those of a model never exposed to the forgotten data, accounting for models' ability to generalize beyond their training examples. This paper's method may fall short in achieving this level of privacy protection.

Questions

- How would you improve the helpfulness of the model after unlearning and metigate privacy risks? - It would be good to compare against other unlearning methods introduced in the literature.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes. limitations were adequately addressed.