Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Summary

This paper presents negative preference optimization (NPO), which is inspired by Direct Preference Optimization (DPO) and uses the forget set as negative responses, and leaves the positive response empty. Authors did experiments on synthetic binary classification and TOFU unlearning tasks and find NPO achieves strong performance. Authors also theoretically prove progression toward catastrophic collapse is exponentially slower than the gradient ascent method.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. Simple, intuitive, and effective method, supported by both theoretical proof and empirical studies. 2. Strong unlearning performance while maintaining model utility.

Reasons to reject

1. Important task vector unlearning [1] baselines are missing. [1] Ilharco, Gabriel, et al. "Editing models with task arithmetic." The Eleventh International Conference on Learning Representations. 2022. [2] Zhang, Jinghan, Junteng Liu, and Junxian He. "Composing parameter-efficient modules with arithmetic operation." Advances in Neural Information Processing Systems 36 (2023): 12589-12610. 2. The motivation for synthetic experiments is not very clear to me, and the connection between synthetic experiments and TOFU experiments is also not very clear. Would be nice to clarify the motivation and why it is necessary in this work.

Questions to authors

N/A

Summary

The paper addresses the problem how to make the model forget after it has been trained on a dataset. Previous methods fail to effectively unlearn due to drastic degradation on the model's general capability/utilities. Inspired by the DPO work, the paper introduces NPO which essentially is a variant of DPO without the positive example. Due to the contrast against a reference model, the training becomes more stable and the resulting model retrains better its original utilities. Synthetic and real experiments demonstrate the empirical effect of the method. Paper also provided rigid theoretical analysis. Paper is well written, easy to read. Theoretically speaking, the introduced method seems to be a slight modification of DPO. Empirically speaking, the method is simple and appears effective to address the problems (eg model utility degradation).

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- Paper is well written and easy to understand. - Method is simple but empirically effective, with experiments support. - Paper additionally provided rigid analysis and proof.

Reasons to reject

- Essentially, the introduced method seems to be a slight modification/specialization of DPO.

Questions to authors

none

Summary

This paper proposes Negative Preference Optimization (NPO), which applies Direct Preference Optimization (DPO) for unlearning certain concepts. It analyzes the differences between Gradient Ascent (GA) and NPO, and shows better performance than GA on the TOFU dataset.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. DPO was appropriately applied to unlearning. 2. Section 2 is comprehensively written and easy to understand. 3. The proposed NPO is a generalization of Gradient Ascent, and it presents evidence of superior performance from the perspective of divergence. 4. The proposed approach demonstrated superior performance than Gradient Ascent on the TOFU dataset.

Reasons to reject

1. The name of the proposed method seems to be misleading. Since NPO does not use y_w, it cannot be considered as preference tuning, which causes confusion. 2. Despite mentioning various previous studies in Section 2, only a few baselines are used as a baseline approach in the main experiment. For example, the experiment doesn’t include the performance of [1]. 3. The models used in the analysis are simple linear layers. It is uncertain whether the same results would be obtained using the Transformer architecture, which is commonly used in LLMs. 4. The validity of the TOFU dataset and the metric used in the main experiment is questionable. The paper proposing this dataset has not been peer-reviewed and is an arXiv preprint. [1] Chen et al., Unlearn What You Want to Forget: Efficient Unlearning for LLMs, EMNLP2023.

Questions to authors

"data already unlearned" contributes to stability. However, during the unlearning process, the randomness in training could lead to a larger gradient size for NPO. How does the size of the gradient relate to stability? Can a larger gradient size be considered unstable? 2. The validity of the TOFU dataset and the metric used in the main experiment: The paper proposing this dataset has not been peer-reviewed and is only an arXiv preprint. The main experiment seems to heavily rely on an arxiv preprint paper. It is reasonable to utilize the experimental setting and dataset which was already published in the other conference or journal such as [2]. 3. There exist no baseline methods in the main experiment. I understand that the main baseline is GA, but I think it is necessary to evaluate the performance of the methods mentioned in Section 2. [2] Jang et al., Knowledge Unlearning for Mitigating Privacy Risks in Language Models, ACL 2023.

Summary

This paper presents an application of direct policy optimization for unlearning knowledge from a large language model (LLM). Experimental results on TOFU demonstrate that the proposed method improves over existing state-of-the-art methods.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

- The paper studies an interesting and important problem. - The paper is well-written. - The experimental findings are interesting.

Reasons to reject

- Analysis over a wider range of datasets in different domains would be very beneficial, as well as conducting experiments with different model architectures

Thank you for your response.

Thanks for the response. It would be good to further describe how the size of the gradient relates to stability in the manuscript. Since most of the questions have been addressed, I revised my evaluation as 6.