Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

Dear reviewer, We would like to follow up on our previous discussion and address any remaining concerns you may have. It appears that the current rating may not accurately reflect your overall perception of our paper. For instance, your review mentioned that "the method is rigorously evaluated through extensive experiments," and your response to our rebuttal noted that you were "impressed by the additional results." These comments suggest a more positive view of our work, yet the current rating does not seem to align with these comments. Since the deadline is approaching, we would greatly appreciate it if you could let us know if there are any remaining concerns preventing you from adjusting the score.

Dear Reviewer uFqn, Thank you for acknowledging that our additional results are helpful. Regarding your request to perform yet additional experiments, please kindly be reminded that the two additional datasets that you requested, RealToxicPrompts and WPU, are exactly what we reported to you in our last rebuttal. Please kindly refer to **W3: ULD performance on other datasets**. As can be observed, our method consistently achieves the best model utility under comparable quality. We would also like to bring to your attention that the current score that you assigned does not seem to match your overall accolade of our paper and rebuttal response, unless we miss anything. Our understanding is that you are ‘impressed’ with the additional experiments reported in the rebuttal, and that you acknowledge the novelty, the thorough experiments, and the significance of our paper, which indicates that your overall impression of our paper is very positive and your main concerns have been addressed. However, a score of 5 that you assigned means that the paper is still of borderline quality. We would love to keep our discussions and continuously improve our paper, but we would also like to request that you consider resolving this inconsistency with a score that fairly reflects the overall high quality of the paper.

We thank reviewer F8Xo for the feedback. Regarding the questions: > **W1: Assistant model requires additional augmented retain data.** We would like to clarify that the fact that our method requires an augmented retain set **does not contradict** our claim that our method is insensitive to the representativeness of the retain set (i.e. resolving challenge #2). In the following, we will explain why this is the case and validate our claims with additional experiments. First, to facilitate our explanation, let us recap the two retain sets required by our method: **Regular retain set**, which contains samples about the retain knowledge; **Augmented retain set**, which contains perturbed samples of the forget data. The augmented retain set serves a very different purpose here. Rather than covering the retain knowledge, the augmented retain set aims to **define the boundary** of the forget knowledge, which is a specific need of our approach. This is because when the assistant method learns the forget knowledge, it can easily generalize to neighboring knowledge. The augmented retain set essentially tells the assistant model to learn the forget knowledge **only**, and not the neighboring knowledge. Therefore, representativeness is not a requirement for the augmented retain set. Also, note that generating the augmented retain set does not even need any real retain data. It just contains perturbed versions of the forget data. On the other hand, it is the regular retain set that needs to represent the retain knowledge, and that would cause challenge #2. Here, we claim that our method is not sensitive to the regular retain set, and **can even get rid of it**. To validate this, we conduct additional experiments on TOFU, where we keep our original augmented retain set but reduce the size of the regular retain set to 75%, 50%, 25%, and 0% of its original size. Results in the following table indicate that this reduction has minimal impact on the model utility of our method. Please note that when the regular retain set drops to zero, the method is very close to your expectation that ‘the assistant model is just trained to memorize the data to be forgotten’ – it does not require any real retain data, just memorizing the forget data and distinguishing from the perturbed forget data. Lastly, we also performed an additional experiment that further confirms that our method performs well on unseen retain data. The results in Table 2 in the attached rebuttal pdf shows that the assistant model has a comparably high entropy (thus flat output distribution) on unseen retain data compared to seen retain data, which indicates that our method is insensitive to the coverage of the seen retain data. We will add the above discussions and experiments to the paper. | TOFU-10\% | Forget quality | Model utility | |------------------------|----------------|--------------| | Target LLM | 2e-19 | 0.62 | | Retain LLM | 1 | 0.62 | | ULD | 0.52 | 0.62 | | ULD-0% regular retain | 0.22 | 0.61 | | ULD-25% regular retain | 0.34 | 0.62 | | ULD-50% regular retain | 0.45 | 0.62 | | ULD-75% regular retain | 0.39 | 0.62 | > **W2: Effectiveness of ULD relies on the augmentation.** We acknowledge that augmented data is crucial for our method. However, as explained in our response to W1, the augmented data should be regarded as an integral part of our method, rather than a plug-in that helps any methods. In fact, the following table, which is a subset of Table 4 in the main paper, shows that adding the same augmented data does not improve the baselines. These results further suggest it is the logit difference that makes the difference, not the augmented datasets. | TOFU-10\% | Forget quality | Model utility | |-----------------------|----------------|---------------| | Target LLM | 2e-19 | 0.62 | | Retain LLM | 1 | 0.62 | | GA+KL | 2e-4 | 0.05 | | GA+KL+augment | 4e-7 | 0 | | | | NPO+KL | 0.07 | 0.32 | | | | NPO+KL+augment | 1e-4 | 0.08 | | | | Offset-NPO+KL | 4e-5 | 0.48 | | | | Offset-NPO+KL+augment | 6e-9 | 0.24 | | | | ULD | 1e-7 | 0.53 | | | | ULD+augment | 0.52 | 0.62 | | |

> **W3: Comparison with task vector and representation fine-tuning methods.** We want to first clarify that our method is **fundamentally different from representation learning** in WMDP. The basic idea of WMDP is to destroy the model knowledge on forget data by minimizing the distance between the model representation and a random vector. By contrast, our method does not destroy the representation but only offsets the forgetting knowledge in output logits with an assistant model. In fact, the principle of destroying forget knowledge is not a new idea. [1] also share a similar principle that destroys the model’s representations on forget data by minimizing the KL divergence of its output distribution with a distribution where the ground-truth token’s probability is manually reduced. Due to time constraints, we cannot do more experiments to compare with the WMDP method. However, we happen to have made the comparison to [1] in our response to other reviewers. Results in Table 1 of the attached pdf show that this method (DI) has worse forget quality and model utility than our method. Second, the task vector method is indeed closer to our method. However, **their rationale for unlearning is different from ours**. They find an unlearning direction in model parameter space, whereas we offset the unlearning knowledge in output logits. Their method may resolve challenge #1 in our paper, but it cannot resolve challenge #2, because simply negating the task vector cannot guarantee that the unseen retain knowledge is not affected. Moreover, most existing works utilizing task vectors only demonstrate its effectiveness on modifying abstract model behaviors (e.g., de-toxic and de-biasing), and few works successfully apply it to unlearn fine-grained factual knowledge [2-4]. To further verify our claims, we compare with the task vector method on TOFU-10%, where we experiment with a wide range of weights for the added vector. Results in the following table show that this method fails to achieve a high forget quality and model utility at the same time, and magnifying the added vector continuously compromises model utility, e.g., model utility drops to 0.36 when the weight is -1.8. | TOFU-10% | Forget quality | Model utility | |------------------|----------------|---------------| | Target LLM | 2e-19 | 0.62 | | Retain LLM | 1 | 0.62 | | ULD | 0.52 | 0.62 | | Task vector -0.2 | 5.3e-19 | 0.6 | | Task vector -0.4 | 1e-15 | 0.58 | | Task vector -0.6 | 8.1e-8 | 0.56 | | Task vector -0.8 | 1.2e-5 | 0.52 | | Task vector -1.2 | 3.3e-6 | 0.48 | | Task vector -1.4 | 1.05e-3 | 0.44 | | Task vector -1.6 | 0.17 | 0.39 | | Task vector -1.8 | 0.24 | 0.36 | We would love to provide more complete experiments. However, due to time constraints, this is the best we can provide to our best efforts. We are confident that our response can answer your concerns, but if it does not, we would greatly appreciate a timely discussion since the deadline is approaching. Thank you very much for your time! [1] Dong et al., Unmemorization in large language models via self-distillation and deliberate imagination [2] Zhang et al., Composing Parameter-Efficient Modules with Arithmetic Operations. [3] Dige et al., Mitigating Social Biases in Language Models through Unlearning. [4] Liu et al., Towards Safer Large Language Models through Machine Unlearning.

Dear Reviewer F8Xo, As the discussion period is about to end, we wanted to check if our response has sufficiently addressed your concerns. Although your review was submitted close to the deadline, we made our best efforts to promptly provide a detailed response with many additional experiments. We would greatly appreciate it if you could review our response and consider re-evaluating our paper based on the rebuttal.