Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Summary

In this paper Local Superior Soups (LSS) is proposed to minimize communication rounds in federated learning (FL) using pre-trained models, specifically tackling data heterogeneity challenges. LSS achieves this by employing sequential model interpolation, maintaining connectivity, and integrating diversity and affinity regularization terms. These innovations enable more local training steps and fewer communication rounds, effectively preventing client drift. Designed for adapting pre-trained models in FL, LSS enhances training efficiency, making it well-suited for deployment in edge computing applications.

Strengths

1. The proposed method can effectively reduce the communication rounds in federated learning (FL) using pre-trained models. 2. The proposed method seems sound. 3. This paper is well written.

Weaknesses

1. Only two small-scale image datasets are used in experiments. More large-scale datasets, especially those in other modalities, should be used. 2. More pretrained models should be explored. 3. More tasks besides image classification should be incorporated into experiments.

Questions

None.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See weakness part

Summary

This paper proposes a method called Local Superior Soups (LSS), a novel technique for model merging in cross-silo federated learning aimed at reducing communication rounds while enhancing model performance. This paper introduces random interpolation, diversity term, and affinity term to alleviate the need for time-consuming model selection and redundant model training. Rigorous experiments on 4 datasets with 11baselines demonstrate the effectiveness of LSS.

Strengths

1. This paper discusses the importance of bridging two low-loss valley to reduce communication rounds. 2. This paper introduces two quantifiable metrics, diversity and affinity, which serve as indicators of model quality.. 3. This paper conducts extensive experiments to illustrate the effectiveness of LSS.

Weaknesses

1. The distinction between LSS and similar federated learning methods such as FedProx, which also incorporates weights from global models to regulate client loss, is not clearly discussed. 2. The subsection 3.3.1 titled "Random interpolation conserving connected low-loss region." lacks mathematical detail to fully understand the interpolation process. 3. The requirement for clients to receive the interpolated model pool ($M$) could potentially lead to significant communication overheads, which may not present a clear advantage over simpler methods like FedAvg or FedProx. 4. The connection between Theorem 3.1 and the core methodology of LSS, specifically the diversity and affinity terms, appears tenuous. These terms do not seem to be directly derived from the theorem, which may weaken the theoretical foundation of the proposed method. 5. This paper should consider referencing relevant literatures or conducting preliminary experiments to support its statments on the part called "Limitation of previous model soups methods".

Questions

1. In Figure 3, how do the results of LSS compare with those of FedProx? Given the similarities in the core concepts between LSS and FedProx, such a comparison would be insightful for evaluating the distinct advantages of LSS. 2. The results presented in Tables 1 and 2 suggest that LSS performs well in the initial training rounds. Can the authors clarify whether LSS is primarily advantageous only during these initial rounds? 3. Furthermore, how should the training process be continued post-initialization? Is it feasible to employ LSS throughout the entirety of the training process, or would alternative methods be more effective in later stages?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please see the weaknesses and the questions.

Summary

This paper proposes LSS, a model interpolation-based local training technique to reduce the number of communication rounds required. The intuition is to regularize local models to connected low-loss valleys, so the aggregated model may have lower loss. LSS is empirically evaluated on a variety of datasets and types of distribution shifts.

Strengths

- Figure 1 and 2, though may not be very rigorous, are clear and provide intuition to the readers. - The proposed algorithm is tested on a variety of datasets and types of distribution shifts.

Weaknesses

- Readability: the notation in section 3 is not clear enough. For example, $n$ is used for both number of data (section 3.1) and number of averaged model (Alg 1), which is confusing. - (minor) It might be a little abuse of notation when using $\mathcal{D}_i$ for both distribution and dataset. I suggest using different notations.

Questions

- Although Figure 2 is intuitive, it might not be so rigorous: high affinity and high diversity do not necessarily guarantee an aggregated model with low loss. In the example, two client’s local models are different “horizontally”, while the major axis of the loss landscape is also horizontal. However, this is not always guaranteed. If the clients models are different “vertically”, the proposed method may just fail. Mathematically, in algorithm 1, the gradient of $dist(f_{p_i}, f_p)$ and $dist(f_{p_i}, \mathcal{M})$ is very likely to be nearly orthogonal. Could you explain intuitively why LSS is expected to perform well in most of the cases? - The scope of this paper is limited to FL with a pre-trained model. I understand that the pre-trained model may be larger in average, which makes convergence more changing. I am curious how LSS is limited to pre-trained models and whether it can generalize to randomly initialized models.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes.

Appreciation for the Positive Feedback and Score Increase

We are truly grateful that our responses have successfully addressed all of your concerns. Thank you for your positive feedback and for raising our score. We deeply appreciate your valuable comments and the time you’ve taken to review our work!