Summary
This paper mainly focuses on the partially class-disjoint data (PCDD) problem in federated learning (FL) settings, which is a common yet challenging problem in distributed data sources. Inspired by a classifier structure (simplex equiangular tight frame, ETF), the authors of the paper propose FedGELA to tackle the PCDD problem. FedGELA is a variant of FedAvg with local model adaptation (personalization): They first define the classifier $W$, which is the ETF that the classifier should converge to. Here, they also take the client local data distribution ($\phi$) into account. Afterwards, the feature extractor $H$ will be optimized locally at each client and communicated via server-client communication. Finally, the global feature extractor and the $W$ at central server, as well as the local feature extractors and the adapted $W$ at clients will be returned.
Strengths
The proposed method is motivated very well. The schematic illustration is also clear. The theoretical analysis is sound. Experiments and the results are good.
Weaknesses
From my understanding, FedGELA focuses only on the alignment in the feature embedding space, which has been done by many previous works (FedGen [1], FedProto [2], …). Therefore this paper lacks significance to some extent. Also, the definition of $W$ (ETF, the global classifier), as well as the locally adapted ones looks straightforward.
Questions
1. You claim that FedGELA could mitigate PCDD from both Global and Local view. But in Algorithm 1, the global model $H^T$ is simply an average of the client model and there is no server-side optimization. Could you please explain this?
2. In Table 1, you claim that FedGELA could mitigate the model skew, could you please explain this point in more details? Also, in terms of “save space”, since you are transmitting the whole feature backbone $H$, which is the resnet18 in your experiments, what is the reason of space saving?
3. Why do you use $H$ to represents 2 different terms, features (Line 114) and global backbone (in the Algorithm)? It’s a bit confusing during reading.
4. Could you shortly explain the converged angles in your figures? Is there a specific meaning of the values? E.g. in Figure 2.
5. In Equation 4, you model the client data skew in the label space via $\phi$, which is based on the number of samples from different classes. Have you experimented with other options? This looks a bit too straightforward to me.
6. FedGen[1] is a method which augment the feature embedding space using a shared feature generator, which could possibly mitigate the issue of “waste of space” by generating synthetic embeddings from minority classes. Could you provide a comparison with this work?
[1] Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. "Data-free knowledge distillation for heterogeneous federated learning." ICML, 2021.
[2] Tan, Yue, et al. "Fedproto: Federated prototype learning across heterogeneous clients." AAAI 2022.
Minors:
1. Line 115, $E_W$ and $E_H$ should be introduced at their first appearance.
2. Line 298, the selection of $E_W$ seems to be dataset specific, is there any default values suggestions?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
FedGELA is only tested in the client data with label skew. Is it also applicable to the data with feature skew?