Federated Learning over Connected Modes

Summary

This paper proposes a method for FL to train a simplex of linearly connected solutions with uniformly low loss. Clients are expressed in the simplex by projecting the gradient onto the simplex and similar clients are close to each other. In each communication round each client samples points in the neighborhood of their projected point and jointly train the solution simplex. Each client is expressed as a point in the simplex based on the gradient update signals. The subregion assignments preserve the similarity of the clients by applying the Euclidean projection onto the positive simples with Riesz s-Energy regularization - this spreads the projections while maintaining the similarity between the clients. As the gradient update signals are informative for the subregion assignment only after the model has been trained for some time, the first stage of the algorithm is FedAvg with simplex learning.

Strengths

The paper is well written and the proposed approach is novel and interesting. The method applies simplex learning to tackle personalized federated learning.

Weaknesses

Some details of the algorithm should be better explained. Some baselines seem to be missing.

Questions

- Could you clarify how simplex learning works in practice? Do you compute the loss and the gradients on each of the endpoints or do you sample one point in the simplex and backpropagate to update the endpoints? On which weights do you compute the loss? I think this is unclear from the paper and it would be beneficial to add a section “Simplex Learning” for the reader. - From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness. I would also see two other baselines/ablations that the authors might want to try to corroborate that this is actually beneficial. First, fedavg simplex learning (no subregions, check global accuracy) and then local fine-tuning for personalization (compute local accuracy) personalization can be done with local simplex learning or standard learning. I guess this could be a way to study the impact of simplex learning by itself and the importance of subregions. - As the authors mentioned in the related work, there are other personalized FL methods that apply simplex learning. Why did the authors not report their results as baseline? E.g. references 29 and 19. - I’m not sure I fully buy the claims and observations in the paragraph on variance reduction (lines 255-272). The main issue with heterogeneity is convergence speed, so to claim some effect on this the authors should show faster convergence on the global accuracy. There is evidence that the effect of the subregions impacts local accuracy, but this is expected as there is better specialization - as it would be with a local fine-tuning stage. - I suggest the authors also have a look at other more recent methods for global learning such as scaffold, feddyn, mime that could help with heterogeneity. They could also be used in conjunction with simplex learning. - I found it odd that the authors decided to show the plots in Figure 2 using the simpleCNN and FEMNIST while from table 1 the cifar10 experiments show more edge of FLOCO over the baselines. - Basically, the method only applies to the last FC layer - did the authors try it on the full model?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are not clearly expanded and should be discussed in a section of the paper.

Summary

In this study, the authors tackle challenges in federated learning by introducing FLOCO, which uses linear mode connectivity to identify a solution simplex in neural network weight space. This approach allows for personalized client model training within the simplex, while also enabling efficient updates to both global and local models.

Strengths

The paper introduces an interesting application of the linear mode connectivity approach in a personalized FL (pFL) setting. This method can be considered an alternative to distance-based and similarity-based pFL algorithms. The algorithm is evaluated under various experimental setups and compared with standard and personalized FL algorithms.

Weaknesses

To strengthen their empirical findings, the authors could benefit from leveraging the bilevel optimization literature for analyzing the convergence of their algorithm.

Questions

$\textbf{Q1}.$ Choosing an appropriate experimental setup is often overlooked in the pFL setting. Note that the following two scenarios are inappropriate for pFL experiments: $\textbf{(1)}$ Clients have enough data to train their local models independently, eliminating the need to participate in an FL system. $\textbf{(2)}$ The data distribution of the clients is identical or nearly identical. This typically results in high accuracy for FedAvg, making it challenging to justify using pFL methods. It would be informative to evaluate your method in experimental setups where local training and FedAvg fail to provide good results and compare your method's performance with SOTA pFL methods, especially methods with decoupled parameter space such as FedPer [1] and FedRep [2] and similarity-based methods such as pFedSim [3]. $\textbf{Q2}.$ As previously mentioned, evaluating the performance of local models on their respective local datasets is crucial in personalized federated learning (pFL). Including the performance metrics of these models (average performance when trained solely on local training data) would provide valuable insights into the experimental results. [1] Arivazhagan, Manoj Ghuhan, et al. "Federated learning with personalization layers." arXiv preprint arXiv:1912.00818 (2019). [2] Collins, Liam, et al. "Exploiting shared representations for personalized federated learning." International conference on machine learning. PMLR, 2021. [3] Chen, Yizhu, et al. "PFedSim: An Efficient Federated Control Method for Clustered Training." IEEE Journal of Radio Frequency Identification 6 (2022): 779-782.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The discussion of the limitations of the proposed method is framed within the context of future directions. It is suggested to address these limitations separately, focusing on theoretical analysis and validation using real-world federated learning datasets suited for personalized FL (pFL).

Summary

A novel method is proposed to derive the mode connectivity over the simplex defined by the central server for an improved global model as well as local personalization performances in the federated settings.

Strengths

The proposed objective using Riesz s-Energy regularization along with the Euclidean projection onto the simplex is novel and convincing. The paper is well structured and various analyses are provided with accessible illustrations. The hyperparameter details provided are helpful in reproducing the results.

Weaknesses

- Line 25: is leverages -> leverages - Line 26, 33, 66, 129: weight parameter -> parameter (since parameters typically represents both weights and biases in NNs) - Line 60: over the clients -> for the $k$-th client - Line 96: (not sure) in the RHS, it may be $\boldsymbol{w}_\boldsymbol{\alpha}$ - Line 127: please consider adding equation numbers in each line of the pseudocode of Algorithm 1. - Line 148: missing citation - Lines 189-190: the baseline methods are not convincing. Since the proposed method explicitly induces the mode connectivity (i.e., line 25 per se) for the improved FL performance, it should have been compared to methods with similar motivations, e.g. [1], which was cited in the Related Works section (line 286) but not directly compared. Plus, authors should have also considered adding more personalization methods (e.g., [2], which shares the same strategy, i.e., only exchanging parameters of classification layer, for communication-efficient personalization in FL) as the authors aim to prove the effectiveness of `FLOCO` for improved personalization performances. - Line 212: please also report standard deviations on 5 different runs. - Lines 461-466: the communication rounds (100 and 500), the number of clients (100), and the number of per-round clients in CIFAR-10 (which is 30 out of 100) are somewhat deviated from the practical FL settings, which raises doubts about the scalability of the proposed method. It is more acceptable if the proposed method is specifically designed for the cross-silo FL settings, which usually assume a small number of stateful clients and (optionally) full participation across rounds. [1] Connecting low-loss subspace for personalized federated learning (2022) [2] Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020)

Questions

- Line 255: in Figure 2, the convergence speed of the global test accuracy & the average local test accuracy of the proposed method is slower than the baseline methods (when seeing round ~50), which is somewhat counterintuitive provided that the total gradient variance remains the smallest across all FL training rounds (which authors stated that it leads to the better performance, i.e., line 265-266). Could the authors please provide an explanation on this? - Line 255: same in Figure 2 (and similarly in Figure 3), the average local test accuracy is suddenly surged, starting from the 200th round. This is quite intriguing, and I also expect an explanation on this result.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors do not evaluate the proposed method out of vision classification task.

Summary

The authors propose federated learning over connected modes (FLOCO), where clients are assigned local subregions in this simplex based on their gradient signals, and together learn the shared global solution simplex.

Strengths

This paper is richer in the type of experiments.

Weaknesses

The proposed methodology lacks innovation. The experimental baseline is too old (2021) and there are many recent and similar clustered federated algorithms that lack comparison.

Questions

1 Can the authors clarify the difference between the proposed method and other clustering federated learning algorithms? 2 Can you specifically describe the implementation details of the algorithm? Or improve the description. For example, there is a lack of more detailed description for the generation of R. 3 In the algorithm section, can the specific algorithm be explained more clearly?

Rating

3

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discuss the reasons for the poor experimental performance of the proposed methods

Extension

***Reviewer’s question 7:*** Basically, the method only applies to the last FC layer - did the authors try it on the full model? ***Author’s answer:*** We show the result with FLOCO-all in Figures 5 and 6 in the rebuttal pdf, where we observe that applying FLOCO to all layers significantly reduces training speed.

Start discussions with authors, please

Dear Reviewers, Thank you for your valuable contributions to the review process. As we enter the discussion phase (August 7-13), I kindly request your active participation in addressing the authors' rebuttals and engaging in constructive dialogue. Please: - Carefully read the authors' global rebuttal and individual responses to each review. - Respond to specific questions or points raised by the authors, especially those requiring further clarification from you. - Engage in open discussions about the paper's strengths, weaknesses, and potential improvements. - Be prompt in your responses to facilitate a meaningful exchange within the given timeframe. - Maintain objectivity and professionalism in all communications. If you have any concerns or need guidance during this process, please don't hesitate to reach out to me. Your continued engagement is crucial for ensuring a fair and thorough evaluation process. Thank you for your dedication to maintaining the high standards of NeurIPS. Best regards, Area Chair