Personalized Federated Learning via Feature Distribution Adaptation

Summary

Federated Learning (FL) combines data from multiple clients to train a global model but struggles with heterogeneous data. Personalized Federated Learning (PFL) creates individual models for each client, addressing this issue. Traditional methods face challenges with bias-variance trade-offs and rely on limited local data or costly techniques. This paper proposes pFedFDA, an algorithm that frames global representation learning as a generative modeling task. It adapts global generative classifiers to local feature distributions, improving performance in data-scarce settings.

Strengths

1. To implement personalized federated learing from generative view looks interesting and promising. 2. This paper is well-structured entirely, but the notations could be improved. 3. The mathematical proof looks good and sufficient.

Weaknesses

1. Your method looks similar to the traditional personalzed federated learning. just as the algorithm 1, the shared backbone is just the weighted summation of the corresponding part in all involved clients. 2. You mentioned that your shared backbone are trained in a generative way, but I can not get the core of of it. Actually, there are some other works[1] use the generative models (e.g., autoregression) as backbone to implement personalized federated learning, what is the difference between your methods and such baselines? 3. Your global distribution parameters are weighted summation of all clients, so in my opinion, the global distribution may be bias due to data size skewness. There are many ways to overcome this, such as [2], etc. do you think such methods can be adopted by you to fix this issue of data skewness? [1] Kou, W.B., Lin, Q., Tang, M., Xu, S., Ye, R., Leng, Y., Wang, S., Chen, Z., Zhu, G. and Wu, Y.C., 2024. pFedLVM: A Large Vision Model (LVM)-Driven and Latent Feature-Based Personalized Federated Learning Framework in Autonomous Driving. arXiv preprint arXiv:2405.04146. [2] Kou, W.B., Lin, Q., Tang, M., Wang, S., Zhu, G. and Wu, Y.C., 2024. FedRC: A Rapid-Converged Hierarchical Federated Learning Framework in Street Scene Semantic Understanding. arXiv preprint arXiv:2407.01103.

Questions

1. "where Φ and H are the feasible sets of neural network and classifier parameters, respectively.", in this context, it looks better to replace "neural network" to "shared backbone", do you think so? 2. The notaions sometimes looks confused, which could be improved. 3. In table 5, it should be pFedFDA not pFedDFA, right? 4. The used dataset looks simple, could you use more complex dataset, such as cityscpaes, camvid, kitti, imagenet, etc to veriry your contributions?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

the proposed method is over-reliance on the local distribution, so it is difficult to handle following cases: 1. a new client is added into the system and its local distribution is far away the original global distribution and the original distributions of all clients; 2. the local distribution of all clients are depondent on time (i.e., dynamic)

Summary

This work introduces pFedFDA, a novel approach to personalized federated learning that conceptualizes global representation learning as a generative modeling task. Specifically, the method involves shared representation learning, guided by a generative classifier characterized by a low-variance global probability density. Additionally, it iteratively refines global parameters and employs a local-global interpolation technique to tailor these estimates to individual client distributions. The authors validate the effectiveness of the proposed solution through experiments on benchmark datasets, demonstrating its robustness and applicability.

Strengths

The paper presents an interesting and meaningful idea, complemented by a theoretical analysis of the bound on high probability estimation errors for the interpolated mean estimate, offering valuable insights and inspiration to the community. The writing is clear and straightforward, with well-designed figures that effectively support and clarify the presented concepts.

Weaknesses

There are several concerns that need addressing: 1.In some cases, the performance improvements appear minimal. For instance, Table 2 shows that pFedFDA is outperformed significantly by FedBABU and pFedME. 2.In Table 3, the authors employ a Dirichlet distribution (Dir(0.5)) for experiments under extreme data scarcity. It would be beneficial to lower the Dirichlet value, perhaps to 0.1, to conduct a more extensive evaluation under greater data scarcity. 3.Table 5 indicates that the runtime of pFedFDA is longer than several other methods, including FedAvg, Ditto, and FedRoD, which calls into question the efficiency of the proposed solution. 4. In Equations 3 and 5, should there be a superscript ‘c’ on Σ to clarify the notation further?

Questions

please check "Weaknesses". I like the idea of this work and I am looking forward to the responses from authors.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper introduces a personalized Federated Learning (FL) method that adapts global generative classifiers to local feature distributions. The authors show that their method can handle complex distribution shifts for computer vision tasks.

Strengths

- The paper proposes a personalized FL method that uses a generative classifier by considering bias-variance trade-off. - The authors conduct extensive experiments. - The paper is well-structured and easy to understand.

Weaknesses

- Sharing the statistics of the generative classifier could increase privacy risks. - The algorithm’s additional computational cost (in Algo.1 L8-9) could increase with larger local data. Calculating the local covariance matrix for larger data and the covariance inversion matrix for each beta value could be computationally intensive.

Questions

- In Fig. 1, the colors representing client 1 and client 2 are not easily distinguishable. - In Eq. (4), wouldn’t the cost of the covariance matrix inversion be high? Could the server calculate it once and send it to the client to reduce the client’s computational cost, even if it increases communication cost? - In Eq. (9), should “\in min” be replaced with “= \argmax”? - If the client’s generative classifier’s mean and variance are sent to the server, wouldn’t the privacy risk increase compared to sending only the weight of the conventional linear classifier? - If the generative classifier is not used and only the weight parameter from the conventional linear classifier is interpolated locally, could the performance be expected to be mid-range between FedAvgFT and pFedFDA? - In experiments like Table 2, where there is less covariate shift between clients within the federation, FedPAC performs better. Could the degree of covariate shift within the federation be determined by looking at the difference in the generative classifier’s statistics coming from the client, and then choose the appropriate FL method?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to the weaknesses section.

Summary

This paper uses Class-Conditional Gaussian Model to formulate the latent representation of the global generative part; for the other part, a personalized federated learning algorithm pFedFDA is designed via Federated Distribution Adaptation. This paper then proves a bound on the bias-variance trade-off of pFedFDA under the assumption of independent and Gaussian distributed local dataset features. And the performance of pFedFDA against dataset scarcity and heterogeneity is presented.

Strengths

1. The introduction and related work are well-organized. The explanation on the replacement of generative global model of the embedding module is convincing. 2. This paper gives a solid proof on the analysis of bias-variance trade-off.

Weaknesses

1. The experiment is concentrated on the CIFAR-10 and CIFAR-100. However, it is also the dataset where pFedFDA has best performance on (Tab. 2). Also, the robustness of pFedFDA against strong heterogeneity is not good. The weakness of pFedFDA against heterogeneity is still a limitation in this paper. 2. The assumption of Theorem 1 is very strong, however, this paper does not give an evaluation on the datasets in the experiment of longtailed-ness or normality. Also, the basic result of Theorem 1 is about the upper-bound of local bias, there is no matric in the experiment to compare the performance of bias-variance balance among the algorithms, especially in the scarcity experiment (Tab. 1, 3). 3. This paper does not explain the connection between bias-variance balance and robustness against data scarcity.

Questions

1. Can the Class-Conditional Gaussian Kernel represent the optimal Gaussian distribution? 2. Further explain the experiment Tab. 1. How can this relate to the advantage of pFedFDA on bias-variance balancing? Also, other models (i.e. APFL, Ditto) have a stable performance enduring the dataset scarcity. How can this be explained? 3. In Tab. 2, the line of FedRep, please double check the value of the last cell .145(0.4). Is that correct? 4. Do you perform normality test on the training set of agents? If not, can you give a further explanation of the feature distribution of datasets to match the assumption of Theorem 1?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. The assumption of Theorem 1 is strong. It would be better to consider the skewed distribution, and even long-tailed distributions. Based on this analysis, the result in Tab. 3 could be expanded to more general few-shot or one-shot learning cases, with a modified version of pFedFDA. 2. The design of pFedFDA is still not robust to heterogeneity and is not capable of better personalization.

Summary

This paper introduces pFedFDA, a personalized Federated Learning (pFL) method designed to address the issue of client heterogeneity in federated learning. pFedFDA combines global knowledge through server aggregation with local knowledge through client-specific training and distribution estimation, enhancing the model's personalized performance on each client. The authors propose an algorithm that efficiently generates personalized models, demonstrating significant improvements in data-scarce settings through extensive computer vision benchmarks. The paper is well-structured, and the writing is clear, making it easy to follow the authors' arguments and experimental results.

Strengths

1.The method of decomposing model training into shared representation learning and personalized classifier training, followed by adaptation to local feature distributions, is both innovative and promising for handling non-i.i.d. data in FL environments. 2.The paper provides strong empirical evidence through comprehensive experiments on various datasets, showcasing the superiority of pFedFDA in challenging distribution shift and data scarcity scenarios. 3.The paper is well-written, with a clear structure that logically progresses from the introduction of the problem to the presentation of the methodology and results.

Weaknesses

1. In this paper, the mean value and covariance are utilized as the characteristic distribution of the data. Have you explored other statistical measures to describe this distribution? 2. In the ablation study, the results obtained by calculating multiple β values were unexpectedly lower compared to using a single β value. Intuitively, multiple β values should provide a more comprehensive understanding of the data distribution and thus yield better results. However, the experimental findings do not support this expectation, and there is no clear explanation for this discrepancy. 3.Note that pFedFDA uses the same feature extractor for all clients but employs heterogeneous classifiers. A key feature of pFL is the heterogeneity of client models. pFedFDA is somewhat limited in this regard. Are there methods to handle heterogeneous feature extractors in pFL scenarios?

Questions

The main concerns and questions are listed in the weaknesses. Please provide answers to address these concerns.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations.