HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning

Summary

This paper identifies that the KD-based method which is used to tackle the data heterogeneity becomes more vulnerable under the model poisoning attacks. Moreover, the models unknowingly align benign client models with a poisoned server model in a malicious setting. It is also called attack amplification. To address this phenomenon, HYDRA-FL is proposed by conducting KD-loss at the shallow layer via an auxiliary classifier.

Strengths

Clear motivation with the case study to present the observation on “model alignment process inadvertently forces local models to align its representation/predictions to the poisoned global model amplifying the attack’s impact”. It is easy to follow up and the HYDRA method is simple to implement which is very important.

Weaknesses

- Only two methods are implemented with HYDRA can not convince me that this method is efficient enough. As these two methods are also outdated, it is better to implement this simple modification to more KD methods. - Depth analysis of experimental observation is needed to provide more insights and inspiration.

Questions

- In FL Settings, Setting a sampling ratio equal to 1 across 10 clients in MOON is not a realistic scenario. - Why only test $\alpha=0.5, 0.1, 0.05$ in the CIFAR-10 dataset, Whats the performance in the other two datasets? - How to explain that the performance gain in a larger $\alpha$ scenario under adversarial setting is slightly low than in smaller $\alpha$. e.g., In Table 1, CIFAR 10 $\alpha=0.05$FedNTD→HYDRA-FL 21.72 to 25.15, $\alpha=0.5$FedNTD→HYDRA-FL 52.51 to 52.57. This phenomenon can also be observed in Table 2 and Figure5.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

See weakness and questions.

Summary

This paper investigated the phenomenon termed *attack amplification* in federated learning with Knowledge Distillation (KD) and proposed an FL framework named HYDRA-FL to reduce the impact of poisoning client attacks in FL. An auxiliary classifier is introduced to employ KD loss on the shallower layer and reduce the effect of KD on the final layers of the model. Experiments on the benchmark dataset demonstrated the performance gain of the proposed method under both clean and poisoning settings compared with a few other FL methods.

Strengths

\+ Showcased a crucial threat of KD, a commonly used technique in federated learning. \+ Provided a unified solution for robust KD that is agnostic to most FL frameworks. \+ Nice ablation study to validate the effects of different components in the proposed algorithm.

Weaknesses

\- Practicality: It seems that this algorithm hinges on careful choices of hyper-parameters of $\beta$, $\mu$, $\gamma$. Sensitivity analysis against drastic changes in these hyper-parameters should be provided. Fig 7 is not sufficient enough. \- Experiments: only two FL+KD methods were analyzed, whereas there are many other related work that employs KD in FL, such as methods that leverage proxy datasets [1]. \- Related work needs to be more comprehensive. For instance, there has been work that discussed KD in FL with poisoned teacher models [2]. Experimental comparison or at least a discussion of work along this line is expected. --- References: [1] Lin, Tao, et al. "Ensemble distillation for robust model fusion in federated learning." *Advances in neural information processing systems* 33 (2020): 2351-2363. [2] Hong, Junyuan, et al. "Revisiting data-free knowledge distillation with poisoned teachers." *International Conference on Machine Learning*. PMLR, 2023.

Questions

* The proposed method seems to focus on model poisoning attacks. I wondered how this algorithm would perform when the local client updates are from models with poisoned data (such as flipped labels). * Authors mentioned that 'completely removing the KD-loss at the output layer may cause a more negative impact than keeping it in a reduced form'. What could be the reasons/implications behind this phenomenon?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Summary

This paper first empirically demonstrates the fact that KD algorithms amplify attack effectiveness. Then the authors propose HYDRA-FL as a method for mitigating the attack amplification problem, through a novel loss function template that can be applied to any FL algorithm wherein the local training objective can be adapted.

Strengths

Very well written. The paper flows nicely and is well organized, and the explanations provided are very clear and thorough. The background concepts are explained well so I think this paper is accessible to a large audience (not necessarily limited to people who have background knowledge in FL). The method is reproducible because the paper explains how exactly they modified publicly-available codes for their implementation. Very extensive empirical testing, including a study that motivates the work and a few ablation studies. The results are presented in very clear figures in addition to being well-explained in the text (including numerical values, qualitative statements, and conclusory statements that summarize the takeaways from each set of results). The proposed algorithm is definitely novel. It is taking (to the best of my knowledge) a completely new approach that therefore has high potential to inspire future work. The paper also does a good job of introducing and justifying the attack amplification problem in a way that could inspire future work.

Weaknesses

This does not seem like a very practical solution for real world settings since it causes a drop in benign accuracy on such simple datasets. I would think that the accuracy sacrifice could potentially be even more significant with larger, more complex models/task. It is not made clear why the accuracy drop is worth the avoidance of attack amplification. It would be helpful if there was some examples/explanation for how this approach is useful in real world settings. For instance, how can attack amplification be problematic in a real world setting and how would HYDRA-FL solve the issue? It is nice that there is some theoretical justification for the attack amplification. The paper could be even stringer if (perhaps in the appendix) you could somehow use this justification to relate to some sort of theoretical justification for the effectiveness of HYDRA-FL. Otherwise, with just the empirical results from a limited array of experimental setups, the claims of HYDRA-FL’s effectiveness are not very sound. It seems like your paper is missing some valuable information, for example, I spent some time trying to understand the threat model, but then I found it in the appendix. So it would be helpful if the main text of your paper referenced the appendix more so that the reader knows what they can go to the appendix to find. The limitations and future work are also easy to miss when just put in the appendix, so I would suggest referencing that appendix in your conclusion in the main text. The related work section isn’t really explaining how your work relates to prior work. It would be nice to have more explanation about prior solutions and what their shortcomings are and how your proposed solution overcomes those shortcomings. This is explained more in the introduction and also later in the paper (e.g. in sec. 4.1), which is helpful, but then reading the related work section it is hard to understand where your work is fitting in. So adding at least a sentence for 2.1.1 or 2.1.2, for example, could be helpful.

Questions

Do you have any hypotheses about how HYDRA-FL would work in general across any type of KD algorithm? It would be nice if you have some justification for why your approach could work more generally and not just with FedNTD and MOON.

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations are addressed in the appendix.

Summary

This work addresses the challenge of data heterogeneity in Federated Learning (FL) and its impact on global model performance. The authors demonstrate why KD is susceptible to the issue of poisoning attacks and use these findings as a foundation to propose a novel method HYDRA-FL. Experimental results demonstrate that HYDRA-FL not only enhances resistance to attacks but also maintains its performance in benign settings.

Strengths

- The authors have studied why Knowledge Distillation (KD) amplifies model poisoning and propose their method based on these findings. - The authors implemented ablation experiments to prove the effectiveness of the proposed method. - The authors will release the codes.

Weaknesses

- The performance advantage is sometimes unstable, such as when alpha=5 in Table 2. The authors could provide a more detailed analysis to explain this variability. - The authors should explain how their findings regarding the amplification of attacks through knowledge distillation are applicable to both standard knowledge distillation learning and KD-based FL, emphasizing the differences and similarities in these scenarios.

Questions

Please refer to the weaknesses mentioned above.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of the work in the Appendix.