FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation

Summary

This paper proposes FedLPA, a novel one-shot federated learning method that uses layer-wise posterior aggregation. It aggregates local models to obtain a more accurate global model without requiring extra datasets or exposing private label information. The key innovation is using layer-wise Laplace approximation to efficiently infer the posteriors of each layer in local models, then aggregating these layer-wise posteriors to train the global model parameters. Extensive experiments show FedLPA significantly improves performance over state-of-the-art methods across several metrics, especially for non-IID data distributions.

Strengths

1. It achieves good performance with only a single round of communication between clients and the server, reducing communication overhead and privacy risks. It performs well on heterogeneous data distributions across clients. 2. FedLPA doesn't need additional public datasets. 3. The paper provides a convergence analysis showing a linear convergence rate for the global model optimization.

Weaknesses

1. The method relies on multiple layers of approximation - empirical Fisher to approximate the Hessian, block-diagonal Fisher instead of full, and approximating global model parameters through optimization. Each approximation introduces some error. These compounding approximations could potentially lead to suboptimal global models. However, the paper's empirical results suggest that in many practical scenarios, these approximations still lead to good performance. Nonetheless, a more thorough theoretical analysis of these approximation errors and their impact would strengthen the paper. 2. Although more efficient than some baselines, FedLPA still requires more computation than simpler methods like FedAvg. 3. Storing and transmitting the block-diagonal Fisher matrices for each layer increases memory usage and communication costs compared to methods that only share model weights. For very large models or with many clients, the increased communication overhead from sharing Fisher matrices could become significant.

Questions

please see the weakness

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

The paper "FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation" introduces FedLPA, a novel one-shot federated learning method that addresses challenges associated with high statistical heterogeneity in non-identical data distributions. The framework uses layer-wise posterior aggregation based on the empirical Fisher information matrix, allowing for the accurate capture and aggregation of local model statistics into a global model. The paper claims that FedLPA improves learning performance significantly over state-of-the-art methods across several datasets without requiring auxiliary datasets or exposing private label information.

Strengths

1. **Originality**: The introduction of layer-wise posterior aggregation using the empirical Fisher information matrix is a novel approach in the context of one-shot federated learning. This method effectively addresses the challenge of non-IID data distributions. 2. **Quality**: The paper provides a rigorous theoretical foundation for the proposed method, including convergence proofs and detailed mathematical formulations. The extensive experimental results on various datasets further support the claimed improvements in learning performance. 3. **Clarity**: The paper is well-structured and clearly written, with thorough explanations of the methodologies and theoretical concepts. The inclusion of figures and tables helps in understanding the experimental results. 4. **Significance**: By improving the performance of one-shot federated learning under non-IID conditions, the proposed method has significant implications for practical applications where data privacy and communication efficiency are critical concerns.

Weaknesses

1. **Implementation Complexity**: The use of layer-wise posterior aggregation and the empirical Fisher information matrix introduces significant complexity. The practicality of implementing FedLPA in real-world settings could be challenging without detailed guidelines or simplifications. 2. **Scalability**: The paper does not adequately address the scalability of the proposed method to large datasets or a high number of clients. Evaluating the computational and communication overheads in such scenarios is necessary to understand the practical feasibility of FedLPA. 3. **Privacy Considerations**: While the method claims to preserve data privacy, the paper lacks a detailed discussion on potential privacy risks and mitigation strategies, which is crucial for federated learning applications. 4. **Experimental Scope**: The experiments, although comprehensive, are limited to a few datasets and simple neural network structures. Additional validation on larger and more complex datasets, as well as a variety of neural network architectures, would provide stronger empirical support for the method's generalizability. 5. **Parameter Sensitivity**: The performance of FedLPA may be sensitive to the choice of parameters, such as the noise levels for the empirical Fisher information matrix. A detailed analysis of parameter sensitivity and guidelines for selecting appropriate parameter values would enhance the robustness of the method.

Questions

I have no question.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have acknowledged some limitations of their work, but further discussion would enhance the robustness of the paper: 1. **Complexity and Practicality**: The complexity of the FedLPA framework may hinder practical implementation. Providing more detailed guidelines or potential simplifications could make the approach more accessible for real-world applications. Additionally, discussing potential trade-offs between complexity and performance would be beneficial. 2. **Scalability**: The paper does not sufficiently address the scalability of the proposed methods in environments with many clients or large graphs. Detailed evaluations of the computational and communication overheads involved in scaling the methods would help understand their practical feasibility. 3. **Privacy Risks**: A deeper analysis of potential privacy risks and mitigation strategies is necessary, particularly in federated learning settings where data privacy is a major concern. Discussing how privacy-preserving techniques can be integrated into the proposed framework would strengthen the paper. 4. **Experimental Validation**: While the experiments are comprehensive, further validation on larger-scale datasets and more diverse GNN architectures would provide stronger empirical support for the proposed methods. Expanding the experimental scope would help demonstrate the applicability of the methods in various real-world scenarios. Overall, the paper makes valuable contributions to federated learning for graph data, but addressing the mentioned weaknesses would further enhance its robustness and applicability.

Summary

This paper proposes a one-shot Federated Learning (FL) method, denoted as FedLPA, to address heterogeneous data distribution among clients. FedLPA does not demand auxiliary datasets or private label information during aggregation on the server side. To achieve this, FedLPA infers the posteriors by leveraging the Fisher information matrix of each layer in local models using layer-wise Laplace approximation and aggregates these to train the global model. Abundant experiment results demonstrate the efficacy of FedLPA compared to conventional FL methods under the one-shot setting.

Strengths

1. The idea of the paper is easy to follow. 2. Instead of measuring co-relations between different layers, the proposed method only approximates layer-wise Fisher to get a good trade off. 3. Extensive experiment results show the superiority of FedLPA under the one-shot FL setting.

Weaknesses

To be honest, I do not buy the one-shot Federated Learning (FL) setting. This setting goes against the idea of FL. But this does not affect my rating. 1. There are some typo problems. 2. Authors should compare the proposed method with differential privacy (DP) FL or prototype-based methods rather than conventional FL methods. All these approaches address communication security but from different perspectives. 3. Would the proposed method also show superior performance on more challenging datasets, like tinyimagnet and office-home? 4. Any proof to show that "computing the co-relations between different layers brings slight improvement"?

Questions

As I mentioned in the weakness, the authors should compare the proposed method with DP FL and prototype-based FL rather than conventional FL methods. I believe the experiments in the paper do not provide a fair comparison. According to the authors in Supplementary Sec.L, the security level of the proposed method is similar to that of FedAVG, meaning it is also vulnerable to attacks targeting FedAVG. Does the only advantage of the proposed method is that attackers have fewer opportunities to strike since they only communicate with the server once?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The topic discussed within the paper is highly related to privacy protection and the authors show a novel method to deal with it.