Summary
This work presents an analysis of targeted forgetting in LLMs via random-label fine-tuning. Random-label fine-tuning is a previously proposed approach to train an LLM to "unlearn" certain tasks: this work presents an analysis of when and why random-label fine-tuning actually results in "unlearning" of a task beyond the data that the model is trained to forget.
The authors fine-tune an a Llama2 7B LLM on 21 NLP tasks with their true labels and then fine-tune the resulting model on those tasks with random-labels. They show that not all tasks are effectively unlearned by random-label fine-tuning: while some tasks achieve random-chance on test data after unlearning, other tasks still generalize well to their test data. They also evaluate "cross-task forgetting", where they analyze the impact of unlearning a task on the generalization of other tasks; here they find, surprisingly, that many tasks are consistently forgotten during unlearning, regardless of the task being unlearned.
The authors then analyze why certain tasks are easily forgotten when others are not. They rule out several possibilities, showing low correlation between forgetting and task difficulty (measured as gold-label fine-tuned accuracy) and that more data does not improve forgetting. They instead find that a model's confidence (the probability of the correct response to the incorrect response) and the variance of the penultimate representations for each task are similarly correlated with forgetting (less confident tasks are easier to forget and tasks with similar representations across inputs are easier to forget).
Next, the authors study the order in which samples are learned vs. unlearned. They show that there is a loose relationship here, finding that samples which are learned later are forgotten earlier during fine-tuning, although samples that are learned early can also be forgotten early.
Finally, the authors present an analysis of how forgetting can affect the information stored within the intermediate representations of a model by training linear probes before and after training to forget. They find that, generally, training a model to forget a task does not significantly diminish the information of that task in the intermediate layers, as linear probes can often still generalize well to tasks after forgetting has occurred.
Reasons to accept
Targeted forgetting is an important topic in LLMs that is still poorly understood and this work takes steps in improving our understanding of it. The main observation that fine-tuning on random labels does not always generalize well across tasks but instead that certain tasks are almost always forgotten during cross-task forgetting is an interesting and important observation.
The work then presents a series of broad analyses that individually improve our understanding of when and why certain tasks are forgotten over others, and what the underlying mechanisms behind forgetting might be.
Reasons to reject
While the initial findings on the generalization of unlearning are interesting and the breadth of analysis of the rest of the work is impressive, I find the final sections (5-7) to be somewhat shallow individually.
More specifically, I think the work makes several interesting observations (certain tasks are always forgotten during unlearning regardless of target task, certain factors are somewhat correlated with unlearning generalization, hard samples are forgotten first, task-specific information is still retained in the layers). However, there is not a cohesive take-away from this analysis, i.e. it is not clear how this information should be used to improve targeted forgetting. I feel that the work would be improved if a depth-first approach to the analysis had been taken, i.e. if more resources had been devoted to understanding why certain tasks are always forgotten, or using the analysis in section 5 to present interventions that could improve forgetting.
As it stands, I think the work is weakened by the fact that the series of findings in 5-7 do not have a cohesive and clear impact on our ability to improve forgetting.