$C^2M^3$: Cycle-Consistent Multi-Model Merging

Summary

The paper proposes a new, cycle-consistent method of merging more than two neural networks in weight space by simultaneously solving multiple permutation-based merge process. The key innovation is to ensure cycle consistency when merging $n>2$ models. The authors showed that the method, together with techniques like REPAIR, can lead to substantial benefits across various architectures and datasets.

Strengths

1. The paper proposes a novel alignment algorithm that, by construction, ensures cycle consistency for merging multiple models. The authors specifically showed the benefit of merging models in a universe space $u$ as a bridge for model permutations. I personally found the idea of a universe space neat and sensible, and this method could be a nice advance in the field of model interpolation & merging. 2. Relatively extensive comparison with existing best methods and across different datasets and settings (including some analysis on the loss landscape, etc.) 3. The paper demonstrated compatibility of the method to adopt techniques like REPAIR which helps maintain good performance (i.e., practical enhancement over other prior methods)

Weaknesses

1. While the concept of a universe weight space is nice, the presumption of neuron permutations and the related theoretical foundations are still based strongly on prior work. 2. I found the paper fail to analyze the pros and cons of the simultaneous global optimization in the main paper (though there are some in the appendix). Specifically, the Frank-Wolfe process will involve a $O(n^2L)$ loop which would be very costly compared to the more "local" approach by Ainsworth et al.

Questions

1. The major concern I have is related to the global optimization process itself for model merging. How "easy" is this optimization process and why conditional GD? How does the optimization scale with wider/deeper networks? I suspect that the empirical observation in Fig. 12(a) in Appendix A.5 exactly suggests the instability in the optimization algorithm. 2. The authors reported the walk-clock time efficiency for merging 2 models in Appendix A.5. How slow is the method for merging $n>2$ models (e.g., say 5, 10), especially compared to setting one model as the "universe space" and perform $n-1$ pairwise optimizations? 3. How exactly is the merging factor $\alpha$ determined when trying to use REPAIR for multiple models in Table 1? 4. While by design the transformation to/from the universe weight space is invertible, but in practice depending on the quality of the optimization problem solving, the actual $P^{AC} \circ P^{CB} \circ P^{BA}$ could in theory still be not so close to the identity matrix. Maybe I missed something, so it'd be great if the authors can clarify.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper further put forward a kind of cycle consistent Multi-Model Merging to merge models after permute simultaneously. It addresses the limitations in previous approaches that only handled pairwise merging. It uses a "universe" space to factorize permutations between models, optimizing all layer permutations simultaneously using the Frank-Wolfe algorithm.

Strengths

- Deterministic result: independent of the random choice of layers - The "universe" space to factorize permutations between models solves issues with previous methods that could accumulate errors when applying cyclic permutations. - This paper has sufficient analyze and discussion on the model width, number of merged models, and linear mode connectivity in the universal basin. - The algorithm to factorize is data-free (based on the Frank-Wolfe algorithm)

Weaknesses

- Sensitive to hyper-parameters - Lack of theoretical guarantee (I personally think there is not much researchers can do in this model-merging sub-domain. Especially in large networks) - Exps are down on small datasets and small networks. Could do some exps on larger network and larger dataset - What is the convergence speed given a different number of parameters? What about comparing to other algorithm such as MergeMany?

Questions

- You mentioned: "the resulting models that we obtain are sensible to a wide variety of factors, from training hyperparameters to the optimization algorithm used" in the paper. Could you please elaberate more on this? - What is your convergence speed comparing to git-rebasin (MergeMany). It would be great to see a Pareto Front showing that given a specific size of set of models, the trade-off between computation time and accuracy and see which algorithm occupies more Pareto Optimal solutions - Is that possible to do some exps on CLIP-based/LLM (like the phi-3 which only has 3B parameters) larger networks? There are bunch of CLIP-fine-tuned networks with the same architecture and LLMs fine-tuned based on phi-3 etc. The exps are done only on EMNIST, CIFAR10 and CIFAR100. The largest network being used is ResNet-16/VGG-16 which are a bit out-of-data nowadays. (I know Git-rebasin was working on VGG-16 and ResNet 20 but still...) - In figure 1.b, can you also calculate the cosine similarity? After all, L2 distance does not make much sense in high dimension space. - The eq 3 is confusing, I am not sure if the author wants to sum p from 1 to n while not equals to q or sum (p,q) pairs from (1,1), (1,2), ..., (n-1,n) while p!=q. I am asking this because the author mentioned "In order to generalize to n models, we jointly consider all **pairwise** problems". But the equation 3 does not show the "pairwise". - What is the meaning of color in figure 3?

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the author have mentioned the limitations well in the appendix.

Summary

The paper introduces Cycle-Consistent Multi-Model Merging for merging neural networks by optimizing neuron permutations globally across all layers, ensuring cycle consistency when merging multiple models. Utilizing the Frank-Wolfe algorithm, this approach addresses inter-layer dependencies and guarantees that cyclic permutations result in the identity map. The method is generalized to handle more than two models by mapping each to a common universe space and thus enhancing alignment robustness.

Strengths

- The paper introduces a method for merging neural networks by ensuring cycle consistency through global optimization of neuron permutations, addressing a limitation in existing pairwise approaches - The use of the Frank-Wolfe algorithm for simultaneous layer optimization and incorporation of activation renormalization demonstrates an innovation and advantages through later experiments

Weaknesses

- While the paper demonstrates the effectiveness of C2M3 across various (simple) architectures and (toy) datasets, it lacks a detailed analysis of the method's scalability to very large models or datasets. A more comprehensive evaluation of performance and computational requirements for larger-scale applications is desirable to understand its benefits - Although the method shows promising results in experimental settings, the paper would be strengthened by including case studies or examples of real-world applications where C2M3 has been successfully implemented. This would help demonstrate the practical utility and robustness. Currently it is primarily focused on classification tasks. - The paper lacks a deep theoretical analysis of the convergence properties and guarantees of the proposed method. Including theoretical insights or proofs regarding the convergence and stability of C2M3 would strengthen the work - Other minor things: - In abstract the number of models is denoted by $N$, while $n$ is used in the main text - Fig 2 does not have "Figure 2" in the caption - the definition 1 is quite ambiguous, stating it as $A\approx B$. However, it is important to clarify the precise meaning of this approximation in a rigorous manner, especially when defining something clearly.

Questions

See the weakness part above

Rating

5

Confidence

2

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes