TIES-Merging: Resolving Interference When Merging Models

Summary

This paper identified two sources of performance degradation when merging fine-tuning models, (i) redundant parameter (ii) sing conflict and proposed TIES-MERGING to improve them.

Strengths

1. The motivation to merge fine-gunning and performance is compelling. 2. Experiments were conducted with both NLP and vision, and the proposed method improved accuracy with both. 3. The proposed method is simple and computationally inexpensive, making it easy to reproduce.

Weaknesses

1. There is no theoretical support for the proposed method. 2. In particular, the effect of sign conflict on performance is nontrivial and requires theoretical support.

Questions

1. Is there any theoretical support regarding sign conflict? 2. I needed help understanding how to see Fig. 6. What is "Parameters With Nonzero Values" in Fig. 6.a mean? 3. In Figure 6, why does it look like you assume that a larger mean magnitude will result in better accuracy? In other words, why do you know that a task vector with a small magnitude does not affect accuracy but also that a task vector with a large magnitude improves accuracy?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

The paper presents a novel method, TIES-MERGING, to merge models in the weight space for multitask learning. It observes an interference problem when linearly interpolating weights, and proposes a simple yet effective two-step solution: parameter trimming for small changes during fine-tuning and sign conflict resolution. The experiments in multitask shows that TIES improves performances, making it a notable (experimental) contribution to the literature of model merging.

Strengths

* The paper's main strengths lie in its simplicity, as highlighted by its clear description and illustration. * The paper successfully builds upon the "task arithmetic" task vector approach to report a new interference phenomenon, and then enhances performance with simple yet important modifications. Model merging is an important topic in multitask, and this paper fills a crucial gap in the current literature. * The experimental framework is robust, with a focus on significant large-scale tasks across CV and NLP domains using recent architectures and fair benchmarks.

Weaknesses

Despite its strengths, some areas require attention. * The contributions, though valuable, are incremental, and the observed gains in multitask learning are consistent but arguably marginal, and the trimming/sign play only a marginal role in this gain. * The experiments focus solely on models trained on different tasks. Yet, weight averaging is also useful to merge models trained on a single target task; on this model soups setup, I speculate that sign interference is less an issue, and that TIES would actually be detrimental as it would increase variance, thus limiting the benefits from combining multiple models, in particular under distribution shifts. As a minimum fix, the title should reflect this specificity, as the current one does not adequately reflect this scope limitation. A (naive) suggestion could be "Resolving Multitask Interference When Merging Models". * Even whithin the multitask challenge, the experiments do not cover two important scenarios. First, multitask as better pretraining before fine-tuning on a target task (as in "Fusing finetuned models for better pretraining"). Second, multitask in the sequential patching scenario (as in "Patching open-vocabulary models by interpolating weight"). * Lack of analysis and clarity of the interpolating coefficient, for TIES and for the baselines. Specifically, without validation samples, could you clarify which values $\lambda$ is used: it seems that it's $\lambda=1/|num tasks|$ for weight averaging, $\lambda=0.4$ for task arithmetic (could you please point where you found this value), but $\lambda=1$ for TIES. Therefore, is the difference between task arithmetic/weight averaging in Table 1 simply due to the use of different $\lambda$? Thus (as further suggested from Table 3), scaling is an important factor, these different values of $\lambda$ hidden the true impact of your core contributions. * Similarly, the ablation study in Table 3 could be made clearer. It is not clear whether the ablations are done one at a time or sequentially all together. If the latter is true, then why do we recover 65.5 on T5-base (when all 4 components are removed), while weight averaging performs 65.9? Moreover, what does it mean to remove "elect" while keeping "disjoint mean"? what does it mean to remove "scaling" (is it $\lambda=1/|num tasks|$ or something different)?

Questions

* What is the impact of task similarity on the number of sign conflicts? Does the number of sign conflicts decrease when two models are fine-tuned on a shared task? * Can you visualize the number of sign conflicts for different trimming ratios? * Can TIES improves downstream transfer learning performances? * Could you plot the curve performance while varying the coefficients given to different tasks: see Pareto curves in "Patching open-vocabulary models by interpolating weight", but also in "Pareto manifold learning: tackling multiple tasks via ensembles of single-task models", a missing yet important related work. * Could you enrich the ablation study to further clarify/highlight the different impacts of the key contributions. * How does ensembling of predictions behave in comparison with weight interpolation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have highlighted in Sections 7.3 and 7.4 that a key limitation is to accurately elect the sign. Yet, the paper would benefit from a dedicated limitation section. It should ideally include that: - weight interpolation lacks proper theoretical understanding, - their approach is only (verified) for multitask learning, - the averageability relies on a large pretraining and "good" hyperparams, - TIES may be less impactful in a "sequential patching" scenario, - they still lag behind the simultaneous multitask learning.

Summary

This paper delves into the challenge of integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. The authors identified that current methodologies overlook the interference that occurs between parameters of different models. This interference can be attributed to two primary sources: redundant parameter values and conflicts in the sign of a given parameter's values across various models. To counteract these issues, the authors introduced a novel method, TIES, which incorporates three key steps: (1) resetting parameters that underwent minimal changes during fine-tuning, (2) resolving conflicts in sign, and (3) merging only those parameters that align with the final agreed-upon sign. The proposed method demonstrated SOTA performance across different settings.

Strengths

- The paper provides a comprehensive analysis of the sources of interference in existing model merging methods, specifically pinpointing redundancies in model parameters and disagreements between parameter signs. This thorough examination underpins the motivation for the proposed method. - The proposed method demonstrates robust performance across a wide array of conditions, including various modalities, domains, task quantities, model sizes, architectures, and fine-tuning settings. This versatility underscores the method's adaptability and broad potential for application - The informative and insightful Section 7, which delves into the significance of different components, particularly the estimation of correct signs during the merge process, serves as a valuable resource.

Weaknesses

- The proposed method is quite heuristic and it would make the paper stronger if the authors can provide more theoretical analysis of the proposed method. - The author did not mention the limitations of their method and potential future work. The authors could further discuss them, which will provide a more balanced view of the method and give readers an idea of the potential directions for future research. - The paper could be improved by discussing more real-world applications of the studied problem, integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. Why is this problem important? This would help to demonstrate the practical value of the method and its potential impact in real-world scenarios. - It would be beneficial to include a sensitivity analysis of the hyperparameters used in the method. Although the authors provided a generic recipe of TIES with fixed hyperparameters, it would be good to see how sensitive the method is to different hyperparameters.

Questions

I don't have further questions for now and I look forward to the authors' enhancements on the weaknesses mentioned above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author did not mention the limitations of their method and potential future work. I have listed some suggestions for improvement in the weaknesses section.

Summary

Merge mutiple **fine-tuned** neural networks, each from an unique task (dataset), into one neural network by weight averaging. Denate the weights of pretrained network as $\theta_0$, fine-tuned network as $\theta_{\tau}$, the weight updating direction $\tau = \theta_{\tau} - \theta_0$ The paper introduces two tricks 1) for each $\tau$ only keep the top $k$% elements in $\tau$ according to their magnitude. (keep the others as zero), 2) during the weight averaging, only retain the dominate direction (+1 or -1) for each elements. The experiments shows the improvements of the proposed Ties-Merging method over other weight averaging methods, such as naive weight averaging, task arithmetic, and regmean. But Ties-Merging is still significantly lagging behind MultiTask training.

Strengths

- The writing is clear and easy to read. - The introduced two tricks are resonable and easy to understand. - Comparing with naive weight averaging, task arithmetic, Fisher averaging, and regmean, the proposed ties-merging performs very well.

Weaknesses

- Even though the proposed Weight Averaging method out-perform other weight averaging methods, it still legs behind MultiTask training. Furthermore, considering both the training cost and inference time, it is hard to see the benefit of the proposed method over mutitask training. As to other baseline methods, such as naive weight averaging, Fisher averaging and regmean, they are designed to solve IID (one fine-tuning task instead of mutiple different tasks) not OOD. These two facts reduce the strongness of the experiments. (But it is fine. Because the two introduced tricks are good enough).

Questions

- Simple averaging is $1/n \sum_{\tau} \theta_{\tau} = \theta_0 + 1/n \sum_{\tau}\tau$ - Task Arithmetic is $ \theta_0 + \lambda \sum_{\tau}\tau$ What is the parameter $\lambda$ of task arithmetic in Figure 5? If it is $0.4$ as line 210, why there is a big gap between simple averaging and task arithmetic in figure 5, ( given that $1/2=0.5, 1/3=0.33$)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

no negative societal impact.

Summary

This paper proposes to resolve the interference of model merging, a solution to combine multiple task-specific models into a single multitask model. It demonstrates two major sources of interference, including redundant parameter values and sign conflict and proposes solutions to resolve the interference.

Strengths

The paper demonstrates some interesting research insights and the solutions are simple and clear.

Weaknesses

1: I question the necessity of Section 7.3. According to the network pruning literature, a high magnitude is always an implication of importance/sensitivity, and a widely used pruning metric. Flipping the sign of or pruning away the Top-k% parameters can cause significant performance drop is well-known to the literature. It seems to have weak connection with this paper’s main argument. 2: I did not find the discussions regarding limitations and future work. 3: More analysis on the experimental results are required. For example, what are the effects of the proposed on each task respectively? Why do some tasks perform better than others?

Questions

Refer to questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do NOT adequately address the limitations.

Response

Thanks for the answer. **Rebuttal 1 & rebuttal 2**: individual-task models do not require additional training, since there are a lot shared models, e.g. Hugging Face. That is problematic in practice. These shared model may not be averageable, due to fine-tuning stragegies. To make two fine-tuned model be averageable, certain mild conditions need to be satisfied in practice, such as small learning rate, mild regularization, etc. So individual-task models still require additional trainings with the proposed method. **Rebuttal 4**: the parameter $\lambda$. Could you provide the exact number of $\lambda$ of different "Number of Tasks" in Figure 5? So that readers and compare the number with "simple averaging" (green line).