Spectral Co-Distillation for Personalized Federated Learning

Summary

The authors proposed a personalized federated learning (PFL) method, where the (dis-)similarity of models are represented through their Fourier spectra. The authors also proposed a bi-directional knowledge distillation between the generic and personalized models and a wait-free local training protocol that can utilize the idle time during global communication so as to reduce the total runtime. Extensive experiments and ablation studies are conducted to demonstrate the performance.

Strengths

1. The authors are the first to characterize (dis-)similarity of models via Fourier spectra in PFL, and the proposed bi-directional knowledge distillation is novel. Further, the proposed simple wait-free local training protocol seems to have the potential of speeding up many existing PFL methods. 2. The paper is well organized and clearly written. 3. The experiment results seem strong.

Weaknesses

The use of Fourier spectra is motivated by previous studies on deep learning theories. As it stands, the method seems like a mathematically motivated heuristic. It would be excellent if certain convergence guarantees or error bounds can be proved, even if just for the most vanilla case.

Questions

1. What are some limitations of this wait-free local training protocol? What kind of methods can adaptively include this protocol into their framework? 2. In the case that a FL system has a large number of edge devices/clients, the participation rate may be low. If I’m understanding correctly, the personalized models keep on training until the updated global weights are received. If an edge device doesn’t participate in training for a long time, wouldn’t this protocol incur a lot of local computation? Also, if the personalized models keep on training without the most updated version of the global weights, is it possible that the personalized models will drift away from the rest of the system?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations. Their work does not seem to have any potential negative societal impact.

Summary

This paper proposes a novel approach to address the challenge of data heterogeneity in personalized federated learning (PFL) by introducing a spectral co-distillation framework based on model spectrum information. The proposed method combines a wait-free local training protocol with bi-directional spectrum knowledge distillation to bridge the training of generic and personalized models. The authors evaluate their approach on CIFAR-10/100 and iNaturalist-2017 datasets using ResNet-18/34 and ResNet-50 architectures, and show that it outperforms existing PFL and conventional federated learning baselines on both personalized and generic model test accuracies. The proposed wait-free training protocol also significantly improves the efficiency of convergence time and has the potential to boost the time efficiency of PFL+ methods. The authors conclude that their approach demonstrates the usefulness of spectral information for knowledge distillation during personalized model training and the potential of using truncated spectral information of personalized models to boost the performance of the generic model.

Strengths

The proposed spectral co-distillation framework is a novel approach that addresses the challenge of data heterogeneity in personalized federated learning. The wait-free local training protocol significantly improves the efficiency of convergence time and has the potential to boost the time efficiency of PFL+ methods. The proposed approach outperforms existing PFL and conventional federated learning baselines on both personalized and generic model test accuracies across multiple datasets and architectures. The authors provide a sensitivity analysis of the truncation ratio τ and demonstrate the usefulness of spectral information for knowledge distillation during personalized model training.

Weaknesses

The proposed approach may require additional computational resources to compute the spectral information of the models, which could increase the overall training time and cost. The authors do not provide a comprehensive analysis of the privacy and security implications of the proposed approach, which could be a concern in real-world applications. The authors only evaluate their approach on a limited set of datasets and architectures, and it is unclear how well it would generalize to other domains and settings.

Questions

see above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see above

Summary

This paper introduces a novel spectral co-distillation framework for PFL, leveraging a Fourier spectrum perspective to measure model similarities. With the assumption that local models share low-frequency features and the global model's Fourier spectrum includes averaged high-level features from all clients, the authors propose a bi-directional knowledge distillation approach to improve performance on both sides. Additionally, a wait-free local training protocol is introduced to reduce runtime. Experimental results on multiple datasets with diverse heterogeneous data settings validate the effectiveness of the framework.

Strengths

**Novel Aspect of Measuring Model Similarity**: The novel frequency perspective for measuring model similarity is intriguing and holds promise for producing a more principled solution for PFL. This aspect deserves encouragement and further exploration. **Interesting Motivation**: The consideration of shared low-frequency components among personalized models and the generic model provides a unique perspective on describing their relationship.

Weaknesses

**Lack of Theoretical Analysis**: The authors highlight in lines 39-41 that existing works lack theoretical support and heavily rely on empirical observations. However, the core motivation behind spectral distillation (line 177) is also intuitive, and no theoretical analysis or guarantees are provided in the paper. This limits the contribution of the proposed method. **Insufficient Experiments**: - The paper would benefit from empirical analysis of model spectra to justify or validate the core motivation. Specifically, it is important to investigate if the robustness of low-frequency components against uniform random perturbation, as demonstrated in Ref[23], remains applicable in the Non-IIDness setting of FL. This would support the assumption of low-frequency component sharing among local models. Providing experimental evidence would solidify and amplify the significance of this assumption. - Empirical analysis for key factors $\lambda_p$ and $\lambda_G$ is missing.

Questions

- The calculation process of the model spectrum is unclear based on the existing descriptions (line 151-155 in the main file, line 73-74 in the appendix). Clarifications are needed, similar to prior work [22][23]. It is better to provide detailed explanations of how the overall model parameters are organized into a single vector and how the spectral vector is calculated. Offering a specific example, such as AlexNet, would greatly enhance the understanding of these processes. - The explanation for the results of the ablation study in Table 5 is inaccurate. Upon comparing the "w/o SCD-PM" line with the "w/o Both" line, it can be observed that solely adopting SCD-GM is detrimental to performance (GM's performance decreases for both $\alpha=1$ and $\alpha=0.1$, while PM's performance remains steady). However, when comparing the "Ours" line with the "w/o SCD-GM" line, the performance improves upon the further adoption of SCD-GM in addition to SCD-PM. This indicates that SCD-PM holds a higher priority and plays a more fundamental role.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

A section specifically addressing the limitations and broad impact is missing.

Summary

This paper proposes a method for PFL, which aims to address the challenge of data heterogeneity. The main idea is to use spectral distillation to capture the relations between the generic global and personalized local models based on the similarity of spectral information. The paper introduces a co-distillation framework that establishes a bi-directional bridge between generic and personalized model training and a wait-free local training protocol that utilizes the local idle time in conventional PFL. The paper evaluates the proposed method on two datasets with heterogeneous settings and shows that it outperforms other existing PFL methods.

Strengths

The paper presents an interesting idea of using spectral distillation for PFL to share low-level features, which differs from existing methods relying on model weight similarity. The paper provides some intuition on why spectral distillation can better capture the generic versus personalized model representations. Furthermore, the paper proposes a wait-free local training protocol that can improve the communication efficiency of PFL, which is a general technique that can be adopted by other PFL methods.

Weaknesses

1. The clarity of this paper should be improved. Please be careful with using uncommon abbreviations, or provide explanations where appropriate. GM and PM should be clarified in the Figure 1 caption; in Line 232, SM should be spelled out as Supplementary Material; In Table 4, what do 3/5 epochs mean? I guess it is the local training steps. 2. Experiment settings need to be improved. The evaluation of PM and GM is not comparable, as they are completely different sets. It would be better to provide additional results of PM and GM on the same test set, such as the global test set. Moreover, the datasets used in the experiments are obtained by splitting with Dirichlet distribution, which are not real FL datasets. Please consider following the experimental setup in pFL-Bench [1], using datasets like FEMNIST [2], and reporting more fairness-related metrics, such as standard deviation. 3. The reproducibility of the experiments is questionable. I could not find how the local test set was generated in CIFAR-10, such as the train/test ratio, in the main text or the supplementary material. [1] pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning, Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Datasets and Benchmarks Track [2] LEAF: A Benchmark for Federated Settings.

Questions

Please refer to the Weaknesses section, and address the issues raised there. I will re-evaluate my assessment of this paper if these Weaknesses are resolved.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

The author discusses the limitation of this paper.

Summary

This paper introduces a spectrum based co-distillation for personalized federated learning; which involves the local model training with regularization/distillation with respect to global parameters's spectral representation and the global model training with regularization/distillation with respect to local parameters's spectral representation.

Strengths

1. Personalized FL is an important and timely topic 2. The approach using spectral transformation for distillation is novel and interesting 3. The experimental approaches show the effectiveness of the proposed approach

Weaknesses

1. The technical contribution is relatively low 2. The experimental results used only two datasets, not very convincing.

Questions

1. Can you highlight both the winner and the second in Table 1? When $\alpha=0.5$, the winner should be FedRod? 2. Why other methods should be affected by $\alpha$ which is a parameter in your model? 3. What are the results if you don't employ spectral based distillation?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

1. This paper proposes a new personal FL methodology: FedSCD. The authors facilitates the generic and personal models to capture global/local distributions through spectral distillation. A co-distillation framework is introduced to help generic and personal models to learn from each other. From the Fourier analysis, since the DNN tends to learn the low frequency features before the high frequency features, the feature level in model representation can be obtained from Fourier spectrum. According to the author, the global model can learn from some low-level features that are commonly shared among diverse local models, and the averaged high level features can benefit the personalized model. 2. Instead of regularizing the local model not to diverge too much from the global model, the spectrum distillation minimizes the loss from the KLDivergence between the spectrum vector of generic and personal model. 3. The paper also designs a wait-free local training protocol that aims to improve the training efficacy by paralleling the process of server aggregation and local personalized model training.

Strengths

1. The idea that utilizing Fourier spectrum information to seek for common global model architecture is novel and properly aligns with the personalization Federated Learning scheme 2. The distillation process, compared to other knowledge distillation methods, does not require public data access on server side, which is more applicable to real-world setting.

Weaknesses

[Methodology Setup and Justification] 1. Co-spectrum distillation design. The design of personal model spectrum distillation is somehow counter-intuitive. In line 56-58, the author says “Dually, the entire Fourier spectrum of the generic model, which includes the "averaged" high-level features across all clients, would benefit the training of the personalized models.” If the low frequency feature is shared among all clients and should be learned by the generic model, the personalized models should have different high frequency features since each clients have distinct data distribution/scenario that they need to adapt on. Then how can the averaged high-leave features benefit client personalization? However, empirically, the spectrum distillation on personal model greatly improves the performance, should such improvement be credited from the fact that during the spectrum distillation, the commonly shared low frequency features in personal model generalized better? So the improvement is actually from better trained low frequency features instead of averaged high-level features? I would like to see author provides more theoretical adjustments on this. 2. Value of Wait-free protocol. The wait-free protocol can be easily implemented under personal FL framework and effectively reduce the communication cost, but it is off-track from the main topic of the paper: co-spectrum distillation. Also, the effectiveness of the proposed wait-free protocol is mainly based on the assumption of all clients share similar training speed. If the fastest client and slowest client has large gap in training speed, the fastest client still need wait for the slowest client. The problem of heterogeneous hardware/computation power is also under explore, and many works have made contribution on this topic[1]. If reducing the communication cost is also a main focus of the paper, the wait-free protocol needs better design to become a new methodology. 3. Missing Fourier Spectrum Analysis in Neural Networks in Related Work. Spectrum analysis of neural networks is apparently a key component in the proposed methodology, but not everyone is familiar with this topic. It will be nice to include a short description of Fourier spectrum in neural networks. --- [Experiment Setting] Overall, the experiment needs better simulation to mimic real settings. The flaws of experiment are below: 1. client participation rate. For all experiment, all clients participate in the aggregation in each round, which is not real in either cross-device or cross-silo scenarios. It is necessary to also show the performance under different client participation rates. 2. Different number of clients in different datasets. In CIFAR10/CIFAR100/iNaturalist-2017, the client number are set to be different(100/50/20) without explanation. More justification on what kind of setting(i.e. cross-silo or cross device) are these experiments trying to mimic on is needed. 3. Other KD related baselines. Since FedSCD borrows the idea of knowledge distillation, the author should also consider to include baselines also incorporate KD such as FedKD[2] and FedMD[3] 4. The different non-iid setting on different dataset. In Table 1, experiment with CIFAR-10, the author shows the result of Dirichlet distribution with alpha = 0.1, 0.5, 1. However, in Table 2, experiment with CIFAR-100, only alpha = 0.1 and 1 are shown. In Table 3, experiment with iNaturalist-2017, only alpha = 0.1 is shown. I would like to see the authors to also make the heterogeneous settings in CIFAR-100 and iNaturalist-2017 as diverse as the setting in CIFAR-10. --- [Theoretical Analysis] 1. Convergence Analysis of spectrum distillation in FL. Spectrum analysis itself has a strong theoretical cornerstone, and at the same time, the methodology of this paper is a significant alteration of traditional FL practice. I would look forward to seeing the authors do some analysis or discussion of the theoretical aspects of the effectiveness of FedSCD, such as the rate of convergence. 2. More intuitive explanation of the accuracy drop in table 5. After removing the PM-SCD in ablation study, although the GM-SCD is applied, the overall performance drops compared to naive setting where both PM-SCD and GM-SCD are removed. It will be nice to provide some intuitive explanation of such accuracy drop. --- Reference: 1. SWIFT: RAPID DECENTRALIZED FEDERATED LEARNING VIA WAIT-FREE MODEL COMMUNICATION 2. FedKD: Communication Efficient Federated Learning via Knowledge Distillation 3. FedMD: Heterogenous federated learning via model distillation ========================Score Modification======================= The author's supplementary experiments and subsequent responses after the rebuttal phase have addressed a number of my concerns. Consequently, I am inclined to elevate my score to a 5. Nonetheless, I still would like to urge the author the author to fully implement the pledged amendments and conduct comprehensive additional experiments for the next versions. Also, I recommend giving further attention to the other issues that I raised yet remain unaddressed such as the theoretical analysis.

Questions

In Table 5, the method without both SCD-PM and SCD-GM share same GM acc with method without SCD-GM and same PM without SCD-PM. But intuitively, since the training scheme is different, the acc should be different. Can you explain this? Please also refer to the Weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

to reviewers

Dear reviewers, Thank you for your service in reviewing for NeurIPS this year! Please carefully read the authors' rebuttal and respond to them by the followings: 1) Acknowledge that you have read the rebuttal 2) Indicate whether and how you plan to change your score (if necessary) 3) Ask the authors follow-up questions (if any) Thanks again! Sincerely, AC