Soup to go: mitigating forgetting during continual learning with model averaging

Thank you for reading the paper and providing a point on novelty that we would be eager to clarify. We want to make it clear that the novelty of our method is not in the pure extension of model merging to multiple domains. Our actual merging method is novel, because it introduces an aspect of averaging that we believe has previously not been discussed: its timing. WiSE-FT (and other common merging techniques) average 2 finished models (whether pretrained and fine-tuned, or 2 fine-tuned, etc.). Our method averages models during training and explores the role of this averaging frequency hyperparameter p. In this way, we introduce averaging as a way to change the course of an active training model’s parameter direction, and in this way connect model merging to classical continual learning methods which rely on a penalty. Furthermore, the unique case of averaging only at the end of fine-tuning (p=1) is equivalent to WiSE-FT. However, as our results show, our method outperforms this baseline, because across all domains, frequent averaging during fine-tuning helps improve overall model performance. Our method performs better than the current baselines, while questioning the existing averaging methods which view it as an after-training technique. We show it can be used before this, during training, to mitigate forgetting while also allowing the model to recover any lost performance on the current task/domain through continued training. We believe this is very novel, yet at the same time intuitive as it builds on existing literature. We completely agree and show that model averaging can be more effective than L2-regression. We wanted to offer this comparison to show that model averaging can approximate L2-regression, rather than completely mimic it. However, we agree that it would be beneficial to show how they differ, and thus explain why one outperforms the other. We updated our paper to include a new version of Section 4, which better shows how SFA differs from L2-regression.

Thank you for showing us where our work was confusing. We would like to clarify this misunderstanding: The update rule we provide aims to break down the L2-regression loss into 2 gradient steps, rather than 2 steps of 1 gradient. We did this in order to mimic SFA (which typically runs a number of gradient steps in the direction of the task loss, before computing an averaging step). As such, Equations 2 and 3 correctly correspond to this by taking 1 step in the direction of the loss, and 1 step that is equivalent to merging the models. As such, we were not attempting to solve the gradient of the original L2-regression. However, we agree that our description can be easily misunderstood, and as such we replaced it with a new Section 4, as well as a Bayesian interpretation for further clarity. We also want to clarify that the entire work is not motivated by equation 3. This section which compares L2-regression with our method is just to provide intuition for why our method works by showing how it approximates L2-regression. Previous work on model merging and our experiments show its efficacy as a technique for mitigating forgetting, where our method naturally extends other merging literature. However, by including this section, we connect model merging to penalty based methods which are potentially more common in continual learning, in a novel way. However, we understand the confusion of this section’s placement, and will move it to the end of the paper so that this is more obvious.

Thank you for your additional response and for helping us improve our paper, we would be eager to clarify further, and address the questions and points in order: - You mention that “In the initial paper, the two gradient steps of SFA are not equivalent to the gradient step of L2-regression, and cannot be regarded as an approximation of the gradient step of L2-regression too. “ We apologize if we have not fully clarified our approach, in our opinion this is not quite what we do. In the initial paper, we take the original L2-regression loss and assume a situation where we perform 2 gradient updates instead of 1: one with the first term, and one with the second. We then show how this 2nd gradient step, which refers to imposing a penalty, can be viewed as model merging. These 2 steps are not 2 gradient steps of SFA, but rather 2 steps from the original L2-regression that have been, in this scenario, split into 2 gradient steps. We do this in order to show how purely imposing a penalty is similar to merging models. In SFA, we take some gradient steps, and then merge models. As such, we create this scenario in order to show the resemblance between SFA and a 2 step version of L2-regression. We think that strictly speaking this is not incorrect. However, we recognize that it can be misunderstood as us claiming equivalence between L2-regression and SFA, as such we updated this in the new version of the paper. - We apologize for not answering this question previously: T is the total number fine-tuning steps on a task. Updating the parameter models corresponds to Equation 4. 1) We have revised the paper to reduce the importance of SFA’s comparison with L2-regression. We did not intend for Section 4 to be one of the primary contributions of the paper. Rather, Section 4 is to provide some intuition for why model averaging might help fine-tuning performance, and not to suggest there is an exact equivalence between L2-regression and model averaging, nor that all of the good performance of SFA can be explained from its resemblance to L2-regression. Nevertheless, it may be helpful because it can provide intuition for why SFA and other model merging techniques work. Specifically, we agree that we take an extreme case of SFA, where averaging occurs after each gradient step, and we compare this to using L2-regression, because it is the closest case of SFA to L2-regression. We show that the weight updates of these 2 cases resemble each other, and can be equivalent under certain conditions. The main justification of SFA lies in its extension of other effective merging methods (Section 3), as well as our experiments where it outperforms existing baselines (Sections 4 and 6). However, we agree that in practice, the benefit of SFA is in its infrequent averaging, and thus while this case shows the resemblance we are trying to highlight, we propose to add a more general case (e.g. averaging after a few gradient steps), to show how SFA moves away from L2-regression as averaging becomes more infrequent. We would be happy to add this to the final paper to add more rigor to the section. 2) We do have L2-regression as a baseline. See Figure 1. 3) As we show in Figure 1, SFA outperforms the baseline of L2-regression. This is consistent with CL literature, and with the contents of Section 4. It was not our intention to suggest that L2 is the gold standard and that any good performance of SFA is due to its approximation of L2-regression. Instead, our intention is to suggest that some intuition behind the good performance of SFA might be explained by its resemblance to L2-regression. However, the fact that SFA outperforms L2-regression, does indeed suggest that there is probably something else at work that might explain performance differences. That is, the way in which SFA differs from L2-regression seems to work in favor of SFA. Nevertheless, we think the comparison is helpful, much in the way that one might explain the performance of L2-regression by first discussing Least Squares.

Thank you for pointing out unclear portions of the paper, we will fix all of these issues. We actually include Llama 2 7B, and a small custom image classification model, in addition to Pythia 2.8B in the experiments, as well as Appendix of this paper to offer differences in scaling and performance. For the final paper, we will also include a more updated model: Gemma 2B, and ViT (see the rebuttal for reviewer kPso). We believe that the drop in performance on GSM8K is important to show how well our method works when catastrophic forgetting occurs. By showing such extremes of forgetting, we only strengthen the effectiveness of our method, because if GSM8K drops to 0%, and our method improves performance so significantly, it must be effective. We have also added a more direct mathematical comparison of our method compared to L2 penalty, as well as Bayesian interpretation, to offer intuition as to why our method outperforms it in the updated version of the paper (Section 4).

Thank you for the comments and insights on our experimental setup. In the current setup, we actually fine-tune on 3 different domains (where each domain can include multiple tasks), on computationally large models and datasets in order to test the robustness of our method. We wanted to highlight these experiments because we show that our method works even in extreme cases as each domain tests very different knowledge and skills. Also, we want to note that fine-tuning LLMs on large language datasets, can be very computationally expensive, and that consequently it is conventional to use a smaller set of tasks than in other domains (e.g. Ilharco et al. 2023 [1] finetune on 4 tasks from the GLUE benchmark). However, we are also currently running and expect to provide more continual learning settings where a single model is trained on a stream of ~20 image classification tasks to show that our method can also be applied to CV with a larger number of tasks. We are currently running experiments using streams of tasks from Food101, and plan to also include CIFAR100 to show that our method works well on continual learning settings in CV. Our experiments tend to show that SFA performs comparably to using data replay, but without the need to store an extra data structure. This in itself is useful given the scenarios we describe where storing a data buffer to use for replay can be expensive to store, particularly because our domains include large datasets, and models. Furthermore, selecting data to store in the data buffer from past tasks, as well as training with a combined dataset of current and past task data can also be an incredibly difficult task. We will fix the spacing issue in the Related Works. [1] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023.