Summary
The authors identify the isolation problem as being the root cause of low accuracy in OFL. The isolation problem arises as the clients locally overfit to spurious features observed in their own local datasets in the absence of knowledge from other clients. To better learn invariant features, the authors propose progressive model fusion called FuseFL which fuses client local models block by block. Crucially, this can be done without increasing the total communication cost, however at the cost of additional rounds.
Strengths
- The authors address an important problem in OFL research in a novel way
- The problem and its solution is clearly presented
- The experimental evaluation is diverse, in particular, the authors also show the memory cost of fusion and experiments with heterogeneous models
Weaknesses
- An important baseline [1] is missing from the evaluations which has been shown to improve over the uniform ensemble considered as upper bound in this paper. The CoBoosting algorithm in [1] generates the weights of the ensemble simultaneously with synthetic data to transfer the knowledge to a single global model and is the recent SOTA. Notably, it is a data-free ensemble distillation method in contrast to Table 1.
- No evaluations are shown with higher heterogeneity levels, such as a = 0.01 or a = 0.05 which is a common setting in OFL papers [1,2,3]. This prevents a comprehensive comparison with prior work.
- The authors have not discussed important limitations of their work. In particular, the security aspect under multiple rounds of communication. OFL algorithms are recognised not just for their low communication costs but also their lower vulnerability to attacks, thanks to the single communication round. By introducing multiple rounds of communication, FuseFL increases vulnerability to attacks.
- Similarly, FuseFL appears to increase client side overheads by letting clients run training K times instead of once as in standard OFL. While the computation cost appears to decrease with K, it is nevertheless higher than client side overheads in standard OFL approaches which mostly add server side overheads. Hence, a comprehensive discussion of these limitations is expected to provide a complete picture of tradeoffs.
[1] R. Dai, Y. Zhang, A. Li, T. Liu, X. Yang, and B. Han. Enhancing one-shot federated learning through data and ensemble co-boosting. In The Twelfth International Conference on Learning Representations, 2024.
[2] C. E. Heinbaugh, E. Luz-Ricca, and H. Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
[3] Jhunjhunwala, D., Wang, S. and Joshi, G., 2024, April. FedFisher: Leveraging Fisher Information for One-Shot Federated Learning. In the International Conference on Artificial Intelligence and Statistics (pp. 1612-1620). PMLR.
Questions
The reviewer is positive about this work and is willing to increase the score provided the authors evaluate against Co-Boosting and include a higher heterogeneity level in their overall assessment. While a thorough empirical assessment of the security of the proposed approach is not deemed necessary, the paper must at least discuss the security aspects along with the client side overheads to better inform the readers of the inherent tradeoffs in FuseFL.
Limitations
No additional limitations than those discussed in the sections above. The authors have well addressed other limitations of their approach involving the memory footprint of the fused model, applicability to model heterogeneity, etc. through their empirical evaluations.