Eliminating Domain Bias for Federated Learning in Representation Space

Summary

This paper addresses the significant challenge of statistical heterogeneity among clients in federated learning. To tackle this issue, the authors propose personalized federated learning, where each client can learn and apply its specific model parameters. Specifically, the authors introduce DBE (Domain Bias Embedding), which comprises two crucial modules. The first module is a client-specific embedding designed to model the domain bias, while the second module is a regularization term incorporated into the objective function to encourage consensus in global representation learning. The effectiveness of these proposed modules is analyzed and demonstrated through theoretical results from domain generalization/adaptation. Moreover, extensive experiments are conducted to verify DBE's efficacy in reducing domain bias and its complementarity with other personalized federated learning methods.

Strengths

1. The paper is well-written, effectively conveying the core idea in a clear and understandable manner. 2. The authors address the persistent issue of non-iidness in federated learning, which remains a significant challenge. The visualization of embeddings pre- versus post-local updates provides valuable motivation for studying and understanding this problem. 3. The proposed DBE (Domain Bias Embedding) approach is both simple and effective, demonstrating strong performance from both theoretical and empirical perspectives. 4. The paper's exploration of the complementary nature of DBE with existing personalized federated learning methods is particularly interesting. This aspect enhances the overall impact and relevance of the paper.

Weaknesses

1. The introduction of the memory module requires improvement. While the module itself is relatively simple, the presentation of its implementation and functioning could be made clearer. It is challenging to understand the process and how to implement it based on the initial explanation. 2. The analysis of generalization risk, although provided, may have limited practical utility when it comes to the practical application of DBE. Further clarification or practical implications would enhance the relevance of this analysis. 3. The experimental evaluation appears to lack diversity in the considered tasks. Including evaluations on tasks such as graph learning would provide additional evidence and make the results more compelling.

Questions

1. What's the difference between the numbers of layers that have memorized domain bias? 2. Is it possible to show that, in the iid setting, the memory embedding is zero?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None.

Summary

This paper tackles the problem of representation bias phenomenon lead by biased domains and proposes Domain Bias Eliminator (DBE), a framework that reduces domain discrepancy between server and client in representation space by Personalized Representation Bias Memory (PRBM) and Mean Regularization (MR). Experimental results show that DBE outperforming ten state-of-the-art methods in both generalization and personalization abilities.

Strengths

- Introduce the PRBM and MR to address the representation bias issue, and the effectiveness of these modules is validated by experimental results. - Provide theoretical analysis on generalization bounds of the global and local feature extractors. - This method has good compatibility and can be combined with other models.

Weaknesses

- It would be nice to evaluate these methods on real-world distribution shifts. - The introduction of MR increases the computational burden. - Some parts of the writing are not clear.

Questions

1. I'm confused about the form of $\bar{z}_i^p$, is it a trainable embedding vector (since it’s the personalized representation of client) or is it a model parameter (as Algorithm 1 claims)? Please elaborate on how $\bar{z}_i^p$ is derived. 2. $\bar{z}^g$ is a consensus obtained during the initialization period, why is $\bar{z}^g$ not updated during the training process? 3. What’s the dimension of the global $z_i^g$ and personalized $\bar{z}_i^p$ implemented in experiments? 4. As mentioned in Line 249 “We run three trials for all methods until empirical convergence on each task”, does this mean that all models are trained until they converge and the communication rounds are different for each model (which is rare in federal learning settings)? And what are the specific conditions for convergence? 5. Figure 1 in Appendix only contains the results of FedAvg+DBE, how about other methods? It would be better to compare the results with other methods to judge the convergence rate. 6. The experimental results of the local model are missing in the paper 7. In Section E.3, the author evaluated the accuracy on 20 new clients. How is the data of these clients divided, and does the class of these data appear in the 80 old clients?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation of the proposed method has been discussed before.

Summary

To address the performance drop via heterogeneous data distribution, the proposed method - Domain Bias Estimator (DBE) - decomposes image features into unbiased (global) and biased (personalized) representations. To guide the feature extractor to obtain the unbiased representation, Mean Regularization (MR) is proposed which is a regularization term for local objective functions that enforce to reduce the gap between mean of representations of local data and a mean of representation of all data across clients. Meanwhile, each client has learnable biased representation which is added to unbiased representation from the feature extractor, and then feed to the local classifier. Unlike the previous works, by considering both unbiased and biased representation, the proposed method could improve bi-directional (local and global) knowledge transfer and could mitigate the performance drop via heterogeneity.

Strengths

- The proposed method handles an important problem of non-IID FL. - The paper is clearly written. - The paper provides theoretical guarantees and comprehensive experimental results that show the efficiency of the proposed method for the given problem. - The proposed method is novel in terms of decomposing representation generated by feature extractor into unbiased and biased representations, and efficient since it requires not much additional computational cost.

Weaknesses

- The paper does not discuss about privacy issues that could be emerged by collection of client-specific mean over local data (line 2,3 in Algorithm 1, Supple). Since the client-specific mean contains representations from all local data, it is potentially exposed to reconstruct identifiable information of local data or even data itself.

Questions

- Collecting averaged features extracted from all local data seems to be exposed to privacy attacks. It would be helpful to give more evidence (previous works or experiments) that the ‘client-specific mean’ collection is privacy-preserving. - Figure 3 shows the role of client-variant (biased) representations and client-invariant (unbiased) representations. However, the representation is not separated according to labels. Are they distinguishable by label for representations extracted by the global feature extractor?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors well describe the limitation of the proposed framework.

Summary

The paper introduces Domain Bias Eliminator (DBE) to address the representation bias and representation degeneration issues commonly observed in federated learning under statistically heterogeneous scenarios. DBE utilizes a Personalized Representation Bias Memory (PRBM) to preserve representation bias and mean regularization to guide local feature extractors toward producing representations with a consensual global mean. The authors provided a thorough theoretical analysis and empirical studies to validate their method. The results indicated that DBE can enhance the performance of existing FL methods.

Strengths

1. This paper has solid theoretical analysis. 2. The proposed method is well-motivated. 3. Comparison with existing methods is sufficient.

Weaknesses

1. The explanation on how DBE reduces the domain discrepancy between server and client in the representation space could be more detailed, as it forms a core part of the technique. 2. The authors could also consider exploring DBE's scalability to larger models, to understand the full potential of the method. The models used for experiments are all small, with only million-level parameters. 3. There are two main hyperparameters in the proposed method and it seems that the recommended choice is quite different for different datasets and models. It is difficult for practical scenarios to tune these hyperparameters.

Questions

Is there any criteria or empirical methods to choose hyperparameters in this work?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations.

Thanks for the response

Thanks for the response and for providing additional representation visualization through t-SNE. Most of my concerns, including privacy issues, have been adequately addressed. I am particularly appreciative of their additional experiment using noisy representation (i.e. client-specific mean). It is intriguing that adding noise to client-specific mean achieves even better accuracy. This is noteworthy, especially considering that introducing noise to gradients typically causes a performance drop. Could the authors provide more insights into the phenomenon? I am wondering if it might be tied to the generalization ability, though I'm uncertain.

Discussions

Thanks for your response! Most of my concerns are resolved. I also read the discussions with other reviewers. It seems that all reviewers are fond of this simple yet effective method. I will keep my score unchanged to support this paper.