DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices

Summary

This paper proposes DapperFL, a novel framework for domain adaptive federated learning designed specifically for heterogeneous edge devices. It addresses the challenge of system heterogeneity and domain shifts in FL by integrating a Model Fusion Pruning (MFP) module and a Domain Adaptive Regularization (DAR) module. The framework also incorporates a novel aggregation approach to handle heterogeneous local models with varying architectures and weights.

Strengths

• Originality: The author focuses on simultaneously addressing system heterogeneity and domain shifts in the context of federated learning. While previous work has devoted to address these two aspects individually, there is a lack of solutions that tackle both problems simultaneously. DapperFL offers a unique combination of model pruning and domain adaptation techniques within a federated learning context, addressing system heterogeneity and domain shifts in a novel way. • quality: The framework is rigorously developed, with a clear methodology and implementation details provided. • clarity:This paper's figures are clear and easy to understand, and the explanations of the framework(MFP, DAR, Aggregation) are well-organized and coherent.

Weaknesses

• The introduction of the two issues, system heterogeneity and domain shift, in this article are not very clear. Regarding system heterogeneity, the author places emphasis on low-capability clients being unable to perform local training, but this emphasis lacks relevance to the rest of the paper. Additionally, the example of the cameras in the domain shifts problem is somewhat biased towards the "multi-view task." For the issues of model heterogeneity and domain shift, the author needs to provide a clearer exposition of the application scenarios where these two problems intersect. If feasible, it would be beneficial to supplement the explanation with illustrations. • The strategy employed in the Model Fusion Pruning phase is not particularly innovative. Moreover, this strategy requires the storage of the entire global model and additional extensive computations under resource-constrained settings, which severely contradicts the problem that this paper aims to address. Furthermore, the pruning strategy employed in this article is the unstructured pruning strategy, so the actual contribution to the problem of system heterogeneity is limited. • Althought the framework diagram is already very clear, it is suggested to include model symbols corresponding to the algorithm in order to provide a clearer explanation of the model's changes at each step. • The method details in this article bear some resemblance to existing work, and there is a lack of references to these works. ( Diao, Enmao, Jie Ding, and Vahid Tarokh. "Heterofl: Computation and communication efficient federated learning for heterogeneous clients." arXiv preprint arXiv:2010.01264 (2020), Yuang Jiang, Shiqiang Wang, Bong Jun Ko, Wei-Han Lee, and Leandros Tassiulas. 2019. Model Pruning Enables Efficient Federated Learning on Edge Devices. CoRR abs/1909.12326 (2019). ) • In addition, there are some approaches that address model heterogeneity and domain shift problems through distillation. Although this paper focuses on pruning paths, these methods should also be mentioned and even compared. (Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. "Data-free knowledge distillation for heterogeneous federated learning." International conference on machine learning. PMLR, 2021. Huang, Wenke, Mang Ye, and Bo Du. "Learn from others and be yourself in heterogeneous federated learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.)

Questions

• Is there a practical difference between the fine-tuning global model steps inspired by the spirit of transfer learning in this article and the local model training after initializing the global model in regular federated learning? Can you provide a more specific explanation of the role of transfer learning in this step? • The global model deployment step appears to result in the global model being stored on the local client and requiring fine-tuning as well as mask operations for traversing each neural unit. Does this contradict resource-constrained scenarios in heterogeneous systems?

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

• The authors pointed out the limitations of the paper in terms of hyper-parameters selection. However, this article still has certain limitations in its solution for system heterogeneity. In addition, it is necessary to add theoretical conversion analysis of the proposed federated learning algorithm.

Summary

The authors propose a new FL scenario that faces challenges of both system heterogeneity and domain shifts. This situation means that the clients have varying capabilities and their data domains also differ. The authors claim that they propose three novel modules to tackle the challenges: Model Fusion Pruning (MFP), Domain Adaptive Regularization (DAR), and a specific aggregation algorithm. The MFP module aims to address the challenge of system heterogeneity. It first fine-tunes the global model with local data to get a local model and then fuses the local model and the global model using a simple weighted average, where the fusion factor decays with the training epochs. Then, it prunes the fused model based on the L1 norm and continues training this model on the local data. The DAR module aims to address the challenge of domain shifts. It adds an L2 regularization on the first few layers of the fused model during the training process. To aggregate the locally pruned models, the models' structure will be discovered using the weights of the global model (of the last communication round) before global aggregation. The authors evaluate their method on two domain generation datasets, and the experimental results show that DapperFL achieves a SOTA performance.

Strengths

1. This paper proposes a novel scenario of Federated Learning (FL) where both the capability and data domains are heterogeneous among clients. 2. Experimental evaluation indicates that the proposed method achieves SOTA performance. This paper also conducts an ablation study to show that each component in the design has a positive effect on the final results.

Weaknesses

1. This paper lacks significant innovation. The proposed framework is just a combination of existing approaches to solving the problems of domain shifts and system heterogeneity, respectively. It’s common to use model pruning to solve the problem of system heterogeneity, such as [1][2][3]. The model aggregation method seems natural for updating the subnetwork of the local model on the global model and has already been proposed in similar work such as [3]. 2. The explanation of the proposed method is unclear. For example, the Domain Adaptive Regularization module does not specify how the model is segmented. 3. The motivation of this paper is not strong enough. This paper lacks evidence to demonstrate the importance of the scenarios present in the Introduction section (with both system heterogeneity and domain heterogeneity). [1] Jiang, Z., Xu, Y., Xu, H., Wang, Z., Liu, J., Chen, Q., & Qiao, C. (2023). Computation and communication efficient federated learning with adaptive model pruning. *IEEE Transactions on Mobile Computing*, *23*(3), 2003-2021. [2] Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., & Chen, Y. (2021, November). Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In *Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems* (pp. 42-55). [3] Li, A., Sun, J., Li, P., Pu, Y., Li, H., & Chen, Y. (2021, October). Hermes: an efficient federated learning framework for heterogeneous mobile clients. In *Proceedings of the 27th Annual International Conference on Mobile Computing and Networking* (pp. 420-437).

Questions

1. How do you segment the model in the Domain Adaptive Regularization module? 2. Do you have any evidence showing the importance or prevalence of the scenario present in the Introduction section (FL with both system and domain heterogeneity)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitation that the proposed method introduces four hyper-parameters and the choice of these hyperparameters significantly impacts the performance of the method.

Summary

This paper proposes DapperFL, an innovative FL framework designed to enhance model performance across multiple domains within heterogeneous FL environments. DapperFL addresses the system heterogeneity challenge through the deployment of a dedicated Model Fusion Pruning (MFP) module. Additionally, a Domain Adaptive Regularization (DAR) module is introduced to improve the overall performance of DapperFL. The evaluation results are positive compared with SOTA.

Strengths

S1: The proposed framework is novel in FL to apply local-knowledge-based heterogenous pruning to the problem of domain shift in FL. S2: This paper achieves better model performance with fewer resource consumptions on a real-world FL platform, which is a significant technical contribution to FL. S3: The paper is easy to follow, with a clear presentation and publicly available source code.

Weaknesses

W1: It could be better to provide a discussion on why only one epoch is enough to fine-tune the global model in Algorithm 1. W2: In the evaluation section, the authors should provide more framework-specific default values for hyper-parameters. W3: The subscript “i” is used to denote both client subscripts in the methods section and client-level subscripts in the experiments section. It would be better to use different notation. W4: The limitations of this work could be elaborated more to let the audience develop a comprehensive understanding.

Questions

Q1: Why is the l_1 norm (rather than other normalization methods such as the l_2 norm) used to calculate mask matrices? Q2: When DapperFL aggregates heterogenous models on the central server, how does the sever know the architecture of these heterogenous models?

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

See weaknesses.

Summary

The paper introduces a novel federated learning framework to address system heterogeneity and domain shifts in edge computing environments. The framework employs a Model Fusion Pruning (MFP) module to generate personalized compact local models and a Domain Adaptive Regularization (DAR) module to enhance performance across multiple domains. Experimental results demonstrate the effectiveness of the proposed method in terms of accuracy improvements and model compression.

Strengths

1. The proposed method is an innovative approach in edge environments. The framework’s ability to simultaneously tackle heterogeneous and low-resource challenges in edge environments. The proposed solution is highly applicable and effective in real-world edge computing scenarios. 2. The paper’s contents are well-organized. The motivation is clearly illustrated and well supported by the proposed method. 3. The experimental analysis is comprehensive. For example, the chosen benchmark datasets include MNIST, USPS, SVHN, SYN, Caltech, Amazon, Webcam, and DSLR. The comparisons with 8 SOTA frameworks add credibility to the claims. 4. The proposed method is technique sounds. Moreover, the source codes are provided to ensure the reproducibility of this work.

Weaknesses

1. The proposed method is sensitive to hyperparameter selection. For example, the framework introduces several hyper-parameters (α0, αmin, ϵ, γ) that significantly influence performance, which also complicates the application of DapperFL. As pointed out in the “Limitations” of this manuscript, it could benefit from providing a set of default values for these hyper-parameters or an automatic hyper-parameter selection mechanism. 2. Experimental Results Analysis: Despite DapperFL achieving competitive model accuracy with a significantly smaller model footprint, it would be beneficial to discuss why a smaller model could outperform a larger one. 3. It is unclear how the proposed meethod to be applied to new devices in test or production phase. 4. Missing some related work, such as heterogenous FL [1,2] and model pruning on devices [3,4]. [1] Zhuangdi Zhu, et al., Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning (pp. 12878-12889). ICML 2021. [2] Yue Tan, et al., Federated Prototype Learning across Heterogeneous Clients, AAAI 2022 [3] Yuang Jiang, et al., Model Pruning Enables Efficient Federated Learning on Edge Devices, IEEE TNNLS 2022 [4] Haiyan Zhao, et al., One-Shot Pruning for Fast-adapting Pre-trained Models on Devices, arXiv preprint arXiv:2307.04365

Questions

1. Any suggestions when choosing hyperparameters (α0, αmin, ϵ, γ) for good global model performance in DapperFL? 2. Why does the smaller model generated by DapperFL have an accuracy comparable to that of SOTA, and sometimes even exceed the accuracy of a full-size model? 3. In eq.6, the authors combine “local knowledge” and “global knowledge” to recover the local model. It could be better to introduce these definitions more clearly before using them. What do “local knowledge” and “global knowledge” in DapperFL stand for, and where are they derived from?

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Response to the rebuttal

Thanks for the authors’ comprehensive rebuttal and the detailed illustrations provided in the pdf file. The responses have effectively addressed my concerns. The clarification on system heterogeneity was particularly helpful. The relevance of DapperFL to the system heterogeneity concern is solved by distinguishing between low-capability clients "failing to complete" training rather than being "unable to perform" it and explaining how MFP mitigates this issue. Additionally, the emphasis on domain shifts and the examples provided have improved the clarity of the scenarios where these issues intersect. My concerns about the suitability of DapperFL is solved by the authors’ clarification on the channel pruning strategy. The authors have mentioned that, DapperFL substantially reduces computation costs in subsequent epochs aside from the minimal and necessary initial costs, which is a significant contribution to the target scenarios. The inclusion of additional references to related work, as well as the reasoning for not focusing on distillation methods, was also satisfactory. Also, I appreciate the decision to include relevant references in the final version. I hope the materials in the rebuttal can be integrated into the final version of the paper to facilitate interested readers’ understanding of the framework. Given the thoroughness and clarity of the responses, I am satisfied that my concerns have been adequately addressed and happy to lift my evaluation of this submission.