Summary
This paper introduces FedMRL, a method based on distillation to mitigate the model heterogeneity issue in Federated Learning (FL). FedMRL operates by learning a small proxy homogeneous global model in a federated manner and distilling knowledge from it to heterogeneous client models. To enhance representation knowledge interaction between the homogeneous global model and the heterogeneous client local model, the authors employ a Matryoshka Representation Learning (MRL) approach, generating multi-dimensional and multi-granular representations. Theoretical analysis and experiments demonstrate the effectiveness of FedMRL.
Strengths
1. The fusion of representations from the global and local models into a single representation vector, followed by their detachment in a Matryoshka manner, is intriguing and inspiring.
2. The writing is good and easy to follow.
3. Transmitting a global model with a relatively lower feature dimension is promising and can reduce communication overhead compared to using a similar dimension as local models.
Weaknesses
1. The authors highlight the limitations of leveraging the training loss between server and client models (incurring high communication and computation costs) and reference the papers FedKD and FML in lines 39-41. However, the proposed FedMRL also falls within this category by sharing a proxy global model. Refer to Figure 4 for an illustration of FedMRL's high communication costs in the MHeteroFL domain, which also conflicts with the statement "low communication costs" in line 68. Additionally, there is a lack of numerical results regarding communication and computation costs between FedMRL and similar methods (FedKD and FML).
2. Two datasets for only image tasks are insufficient in FL.
3. The client models utilized in the experiments are not sufficiently heterogeneous, as they consist of CNN networks with identical numbers of Conv and FC layers. The variations are limited to the channels in Conv2 and the neural count in FC1. This setup lacks the persuasiveness needed to demonstrate FedMRL's effectiveness in MHeteroFL, especially considering that model architectures can significantly differ in size and structure, as noted in [1]. Can FedMRL accommodate settings involving CNNs and Vision Transformers (ViTs) on clients? Additionally, the considered CNNs are overly simple and small for a comprehensive evaluation.
4. There is only one baseline for the model split category, and it's worth noting that FedGH[2] also falls within this category.
5. The details of computing FLOPs are missing.
6. In the "Proof of Theorem 2" section, obtaining Eq. (31) directly from Eq. (30) is not feasible, as the right side is $\frac{\Delta}{T}$, not 0 as suggested by Eq. (30). Additionally, the existence of solutions for $\eta$ in Eq. (32) may be compromised if $\epsilon < \delta^2$, contradicting the conclusion in line 458 and potentially undermining the convergence guarantee.
7. The privacy analysis presented in lines 486-490 lacks sufficient substantiation. It would benefit from either theoretical analysis or experimental results. Without further analysis, it's challenging to accept the claim that "representation splicing enables the structures of the homogeneous global model and the heterogeneous local model to be not related," especially considering that the global and local models are trained together with a shared representation projector.
[1] Zhang, Jianqing, et al. "Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.
[2] Yi, Liping, et al. "FedGH: Heterogeneous federated learning with generalized global header." Proceedings of the 31st ACM International Conference on Multimedia. 2023.