Federated Learning via Meta-Variational Dropout

Summary

The submission proposed a novel Bayesian meta-learning approach metaVD for federated learning. metaVD learns to predict client-dependent dropout rates via a hypernetwork, helping address the model personalization and limited non-i.i.d. data problems. At the same time, metaVD compressed the model parameter, alleviated the overfitting and reduced the communication costs.

Strengths

MetaVD encompasses a new posterior aggregation strategy to consolidate local models into a global one. In addition, MetaVD predicts the dropout rates of parameters via a hypernetwork, enabling parameter compression. This not only allievates the overfitting problem but also reduces the communication costs of exchanging model parameters.

Weaknesses

In general, this article is written in a fluent, simple, and easily understandable manner. However, there are two main issues that need to be addressed: 1. The proposed method in the article is straightforward and intuitive, but many aspects lack theoretical guarantees and analysis. 2. In the past years, Bayesian federated learning has made significant progress, but many relevant works have not been discussed or compared in the article. 3. if my understanding is right, the propsoed method has the risk of data leakage.

Questions

1. “Recently, the Bayesian learning paradigm was introduced to the FL to tackle overfitting by considering the uncertainty of the model parameters [28–30]. However, they could also struggle with diverging local models if the data from different clients exhibit significant statistical variability.” This statement is incorrect. In recent years, Bayesian federated learning has made significant advancements. Many Bayesian methods have been employed not only to address overfitting and introduce uncertainty but also to tackle personalized scenarios. I list some works as reference below. 2. "Moreover, previous Bayesian FL methods are not developed for model personalization, and the performance can be degraded in non-i.i.d. client data." This statement is incorrect for the above reasons. 3. Why was the variational distribution in Equation 3 designed in such a form? I understand that this is an engineering approach, but can this parameterized form effectively approximate the posterior distribution of the weights? Is there any theoretical analysis available? Furthermore, why is the variance parameter $\alpha_m$ used to model the dropout rate? Is there any inherent connection between the two? While this may be an engineering approach, is there any theoretical or intuitive analysis supporting it? It is important to note that while engineering approaches are often driven by empirical performance, they can also be guided by theoretical insights or intuitions. 4. In eq(5), what is the definition of $g_m$? How does the expression of $r_k^m$ come out? Any derivation? 5. In the server-side optimization, when you update the parameters $\psi,e^m$, you are trying to optimize the $L_{ELBO}^m$ w.r.t. $\psi$ and $e^m$, but $L_{ELBO}^m$ requires the data on each client, doesn't this imply data leakage? 6. In the experiment section, as I mentioned earlier, this article lacks thorough investigation into Bayesian federated learning (FL), as it fails to mention and compare many Bayesian FL approaches specifically designed to address personalized scenarios. To name a few examples but not exaustive: [1] Personalized federated learning with gaussian processes, NIPS 2021. [2] Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process. H Yu et al. 2022. [3] Personalized federated learning via variational bayesian inference, ICML, 2022.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discussed the limitations of their work, but they did not address the potential negative societal impact it may have.

Summary

This paper proposes a new federated learning (FL) framework to address the various issues of FL. Specifically, this framework (a) leverages Bayesian FL to address the issue of non i.i.d. data among clients, (b) instantiates its BFL model with a variational dropout posterior to efficiently handle large amount of clients, and (c) applies meta learning adaptation strategies (e.g., MAML and Reptile) at client-sides to personalize the local models. The paper demonstrates the performance of this approach on several vision datasets, with ablation studies to confirm the effectiveness of proposed components.

Strengths

The proposed method makes sense and addresses various relevant problems in the field of FL. I think the paper did an excellent job combining various ideas, and presenting them as one coherent framework. In addition, the empirical results positively improve over existing baselines, which suggests strong practical merits. I also like that the paper is really rigorous with conducting ablation studies.

Weaknesses

The most apparent weakness of this paper is the lack of technical novelty. Bayesian FL, Variational Dropout, and Meta-learning are all very well-known strategies. While I am not opposed to a creative combination of ideas, such idea should have a scientific merit that outweighs the sum of its individual components. Here, each component is behaving exactly as expected and there seems to be no technical challenge in terms of integrating them. I think the novel adjustment here is the hypernetwork (although I am not sure if this strategy has been previously adopted in the context of variational dropout --- e.g., hierarchical prior is sort of similar, but not exactly the same). Some other weaknesses: - Lack of error bar/standard deviation in any of the reported results. - I would prefer some sort of performance vs. communication rounds to demonstrate convergence. Result tables are acceptable, but very uninformative. - It was previously claimed that the hypernetwork helps with large number of clients, especially when some of them have limited data. I think this should be supported by an ablation study showing the effect of increasing no. clients/ decreasing client data (the current ablation study only compares MetaVD to NaiveVD/EnsembleVD on the default client setting). Some other minor issues: - Various typos (e.g., FEMINIST, line 187; Hetrogenity, table 2 header ...)

Questions

- What is the benefit of the hyper-network, as opposed to directly learning the client-specific dropout rates ? - What is the overhead computation cost of the proposed framework? I believe both local meta-learning and training a server-side hypernetwork is quite costly, and it is quite unfair to compare 1000 comm rounds of MetaVD to 1000 comm rounds of FedAvg.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have discussed some limitations of the work in the conclusion. I think the method is purely theoretical and there is no potential negative societal impact.

Summary

This paper proposes a novel Bayesian personalized federated learning approach using meta-variational dropout. The proposed approach employs a shared hypernetwork to predict the client-dependent dropout rates for each model parameter, enabling effective model personalization and adaptation in the limited non-i.i.d. data environment. The effectiveness of this approach is demonstrated empirically on various FL datasets, including CIFAR-10, CIFAR-100, FEMINIST, and CelebA and multi-domain FL datasets.

Strengths

1. The paper is clearly written and well organized. 2. The proposed method in the paper is well-motivated and technically correct. 3. The Bayesian approach to FL is interesting and seems suitable for personalization. The introduced MetaVD seems to work well in practice. 4. The method is extensively tested on a variety of FL datasets.

Weaknesses

1. The discussion with related work needs to include other Bayesian treatments for personalized FL, e.g. [1], [2] 2. The experiments lack a comparison with relevant methods mentioned above [1-2], as well as the baseline in [3]. [1] Personalized Federated Learning via Variational Bayesian Inference. ICML 2022. [2] Fedpop: A Bayesian approach for Personalised Federated Learning. NeurIPS 2022. [3] Personalized Federated Learning using Hypernetworks. ICML 2021.

Questions

1. Page 3, line 115: i.e., $\phi=\phi^1,..,\phi^M$ should be $\phi=\{\phi^1,..,\phi^M\}$. 2. Page 3, line 132: add a definition of $\mathbf{1}$ for $\epsilon^m \sim \mathcal{N}(\mathbf{1}, \alpha^m)$ . 3. In equation (5), how was the aggregation rule derived? It is recommended to provide more details (referring to [4] is possible). 4. It would be clearer to specify the specific sections of the Appendix. For example, on page 6, "More details are in Appendix." 5. The sparsity in Table 7 seems to only consider the personalized layer (the last fully connected layer). I am curious about the proportion of discarded parameters to the total parameters of the model. 6. I hope the authors can provide a discussion on the communication complexity of the additional hypernetwork. 7. Algorithm 1 seems confusing. It would be clearer to separate Algorithm 1 into two distinct algorithms. [4] A Bayesian Federated Learning Framework with Online Laplace Approximation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA.

Summary

The paper introduces a novel approach called meta-variational dropout (MetaVD) for addressing challenges in federated learning (FL). Traditional FL faces issues such as model overfitting and divergence of local models due to non-i.i.d. data across clients. MetaVD leverages Bayesian meta-learning to predict client-specific dropout rates using a hyper network. Extensive experiments conducted on various FL datasets demonstrate that MetaVD achieves good performance.

Strengths

1. The proposed approach is novel. 2. Authors performed experiments on a variety of tasks and datasets and showed promising results. 3. The proposed method showed consistent improvements over the considered baselines with a good margin.

Weaknesses

A primary weakness is that the authors have not compared their results with existing state-of-the-art personalized FL algorithms, e.g., [1,2] and the related work/baselines referenced within them. Other notes and requests for clarification: 1. In L66, where the authors discuss the convergence guarantees of the original FL algorithm, it is important to provide a citation. 2. The authors have written about the “Challenges of FL” in Section 2. They should briefly describe how the proposed approach addresses these challenges. 3. Figure 1 requires additional details to aid understanding. The authors should clarify the meaning of the "x" in the modules of the figure and explain how it illustrates the difference in aggregation between MetaVD and FedAvg. 4. In Table 3, the signs of "+" and "-" should be reversed to ensure accurate representation. 5. Table 4 should explicitly mention that the results presented are for out-of-distribution (OOD) clients to provide a clear context for the findings. 6. Figure 1 is not referenced or discussed in the paper. It should be either referred to in the main text or removed if it does not contribute significantly to the paper's content. 7. Additional information regarding the computation of gradients for the hypernetwork parameters should be included. 8. The authors should provide an explanation as to why the PerFedAvg+MetaVD+DP combination in Table 7 leads to improved performance despite dropping almost 80% of the parameters. 9. Regarding equation 5, the authors should elaborate on the implications of the inverse dependence of the aggregation weights on the square of the model weight. This information would enhance the understanding of the aggregation process and its impact on the final results. [1] “Fusion of Global and Local Knowledge for Personalized Federated Learning”, TMLR 2023. [2] “FedALA: Adaptive Local Aggregation for Personalized Federated Learning”.

Questions

(see "weaknesses" above).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors haven’t discussed the limitations of their work.