Library

Hierarchical Federated Learning with Multi-Timescale Gradient Correction

While traditional federated learning (FL) typically focuses on a star topology where clients are directly connected to a central server, real-world distributed systems often exhibit hierarchical architectures. Hierarchical FL (HFL) has emerged as a promising solution to bridge this gap, leveraging aggregation points at multiple levels of the system. However, existing algorithms for HFL encounter challenges in dealing with multi-timescale model drift, i.e., model drift occurring across hierarchical levels of data heterogeneity. In this paper, we propose a multi-timescale gradient correction (MTGC) methodology to resolve this issue. Our key idea is to introduce distinct control variables to (i) correct the client gradient towards the group gradient, i.e., to reduce client model drift caused by local updates based on individual datasets, and (ii) correct the group gradient towards the global gradient, i.e., to reduce group model drift caused by FL over clients within the group. We analytically characterize the convergence behavior of MTGC under general non-convex settings, overcoming challenges associated with couplings between correction terms. We show that our convergence bound is immune to the extent of data heterogeneity, confirming the stability of the proposed algorithm against multi-level non-i.i.d. data. Through extensive experiments on various datasets and models, we validate the effectiveness of MTGC in diverse HFL settings. The code for this project is available at \href{https://github.com/wenzhifang/MTGC}{https://github.com/wenzhifang/MTGC}.

cs.LGPrivacy-Preserving Technologies in DataStochastic Gradient Optimization TechniquesBrain Tumor Detection and ClassificationComputer Science
View graphSource

Paper

References (58)

01FedBiOT: LLM Local Fine-tuning in Federated Learning without Full ModelFeijie Wu, Zitao Li, Yaliang Li et al.2024 · Knowledge Discovery and Data Mining · 144 citations In Library
02Federated Optimization with Doubly Regularized Drift CorrectionXiaowen Jiang, Anton Rodomanov, Sebastian U. Stich2024 · International Conference on Machine Learning · 18 citations In Library
03Taming Subnet-Drift in D2D-Enabled Fog Learning: A Hierarchical Gradient Tracking ApproachEvan Chen, Shiqiang Wang, Christopher G. Brinton2023 · IEEE Conference on Computer Communications · 10 citations In Library
04Communication-Efficient Federated Optimization Over Semi-Decentralized NetworksHe Wang, Yuejie Chi2023 · IEEE Transactions on Signal and Information Processing over Networks · 8 citations In Library
05Optimizing the Collaboration Structure in Cross-Silo Federated LearningWenxuan Bao, Haohan Wang, Jun Wu et al.2023 · International Conference on Machine Learning · 42 citations In Library
06A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging2023
07Local Exact-Diffusion for Decentralized Optimization and LearningSulaiman A. Alghunaim2023 · IEEE Transactions on Automatic Control · 43 citations In Library
08Decentralized gradient tracking with local stepsYue Liu, Tao Lin, Anastasia Koloskova et al.2023 · Optim. Methods Softw. · 68 citations In Library
09Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated LearningG. Malinovsky, Kai Yi, Peter Richtárik2022 · Neural Information Processing Systems · 51 citations In Library
10Anchor Sampling for Federated Learning with Partial Client ParticipationFeijie Wu, Song Guo, Zhihao Qu et al.2022 · International Conference on Machine Learning · 28 citations In Library
11ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!Konstantin Mishchenko, G. Malinovsky, Sebastian U. Stich et al.2022 · International Conference on Machine Learning · 211 citations In Library
12On the Convergence of Clustered Federated LearningMa Jie, Guodong Long, Tianyi Zhou et al.2022 · arXiv.org · 64 citations In Library

Scroll for more · 38 remaining

Similar papers

GC-Fed: Gradient centralized federated learning with partial client participationJungwon Seo, Ferhat Özgür Çatak, Chunming Rong et al.2026 · Information Fusion In Library
GC-Fed: Gradient Centralized Federated Learning with Partial Client ParticipationJungwon Seo, Ferhat Ozgur Catak, Chunming Rong et al.2025 · Information Fusion · 4 citations In Library
Local Gradient Regulation Stabilizes Federated Learning under Client HeterogeneityPing Luo, Jiahuan Wang, Ziqing Wen et al.2026 · arXiv.org · 0 citations In Library
Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient CompressionZhida Jiang, Yang Xu, Hongli Xu et al.2023 · IEEE Conference on Computer Communications · 47 citations In Library
Convergence-Guaranteed Federated Learning through Gradient Trajectory Smoothing with Triple-Objective DecompositionHaoran Zong, Xiao Zhang, R Q Li et al.2025 · ACM Transactions on Knowledge Discovery from Data · 2 citations In Library
CPFedAvg: Enhancing Hierarchical Federated Learning via Optimized Local Aggregation and Parameter MixingXuezheng Liu, Yipeng Zhou, Di Wu et al.2025 · IEEE Transactions on Networking · 8 citations In Library
FedLGA: Toward System-Heterogeneity of Federated Learning via Local Gradient ApproximationXingyu Li, Zhe Qu, Bo Tang et al.2023 · IEEE Transactions on Cybernetics In Library
Adaptive Control of Client Selection and Gradient Compression for Efficient Federated LearningZhida Jiang, Yang Xu, Hongli Xu et al.2022 · arXiv (Cornell University) · 14 citations In Library
Towards Optimal Multi-Modal Federated Learning on Non-IID Data with Hierarchical Gradient BlendingSijia Chen, Baochun Li2022 · IEEE Conference on Computer Communications · 69 citations In Library
Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient HarmonizationXinyu Zhang, Weiyu Sun, Ying Chen2023 · IEEE Signal Processing Letters · 17 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer zFr16/10 · confidence 3/52024-06-27

Summary

This paper proposes a multi-timescale gradient correction (MTGC) methodology to deal with multi-timescale model drift. It introduces a distinct control variables to correct the client gradient towards the group gradient, and correct the group gradient towards the global gradient. Then, the stability of the proposed algorithm against multi-level non-i.i.d. data is shown empirically.

Strengths

1. The idea of correcting gradients is interesting 2. The proof seems correct.

Weaknesses

The model is tested on small datasets.

Questions

1. What are the practical applications of HFL? 2. Are there any techniques implemented for preserving the privacy of the model? 3. The model has been tested on small datasets; how does it perform on larger datasets? 4. The ablation study is missing. How does the model perform when only one of the corrections is used? 5. How about the stability of the proposed method?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Authorsrebuttal2024-08-11

Thanks for your feedback. We appreciate your time for the discussion. We agree with the reviewer that preserving privacy of clients is crucial in FL. Here, we would like to emphasize that implementing additional privacy-preserving techniques on top of each scheme would degrade not only the accuracy of our approach, but also that of the baselines (they also were not integrated with privacy-preserving techniques in their original work). To demonstrate this, we have conducted an experiment to compare the performance of our algorithm with HFedAvg under the same $(\epsilon, \delta)$-differential privacy (DP) guarantees. We apply DP to the gradient to ensure sample-level privacy. To implement DP, we employ gradient clipping followed by the addition of Gaussian noise, a standard practice in the literature [R1, R2]. This approach is applied to both our algorithm and the baseline. Specifically, to guarantee DP, we replace step 7 of Algorithm 1 in the manuscript with the following procedures: + Compute stochastic gradient: $g_{i,m}^{t,e,h}, m=1,\ldots B_s$, $\forall i $ + Gradient clipping: $\hat{g}\_{i,m}^{t,e,h} = g_{i,m}^{t,e,h} \cdot \min\left(1, \frac{c}{||g_{i,m}^{t,e,h}||}\right), m=1,\ldots B_s$, $\forall i $ + Applying Gaussian noise to the gradient: $G_i^{t,e,h} = \frac{1}{B_s} (\sum_{m=1}^{B_s} \hat{g}_i^{t,e,h} + \mathcal{N}(0,\sigma_g^2))$, $\forall i $ + Local model update: $ \boldsymbol x_{i,h+1}^{t,e}= \boldsymbol x_{i,h}^{t,e} - \gamma( G_i^{t,e,h} + \boldsymbol z_i^{t,e} + \boldsymbol y\_j^t), \forall i \in \mathcal{C}\_j,\forall j$ To achieve $(\epsilon, \delta)$ privacy, the noise should be set at the following scale [R1, R2]: $$ n \sim \mathcal{N}(0,\sigma_{g}^2) ~~ \text{while} ~~ \sigma\_{g}^2 = \frac{c^2 \log (1 / \delta) THE}{(|\mathcal{D}\_i|/B_s)^2 \epsilon^2}, $$ where $B_s$ denotes the batch size. The table below shows the results using the Shakespeare and CIFAR-100 datasets. For the Shakespeare task, we set $T=30, E=30, H=35$, $c=1$, $\delta = 10^{-3}$, $|\mathcal{D}\_i| = 1500$, and $B\_s = 200$. We compare their performance under a privacy budget of $\epsilon = 15$. The standard deviation of the Gaussian noise is thus $\sigma_{g} = 1.37$. For CIFAR-100, we set $T=50, E=20, H=20$, $c=10$, $\delta = 10^{-3}$, $|\mathcal{D}\_i| = 500$, and $B\_s = 50$. Under the privacy budget of $\epsilon = 15$, the standard deviation of the Gaussian noise is $\sigma\_{g} = 1.63$. | Dataset | MTGC (w/o DP) | MTGC-DP | HFedAvg (w/o DP) | HFedAvg-DP | |-------------------|-------|---------|---------|------------| | Shakespeare | 46.42 | 45.50 | 43.16 | 42.95 | | CIFAR-100 | 53.53 | 50.72 | 41.69 | 39.70 | We see that, as expected, the noise injection process for guaranteeing DP slightly decreases the accuracy of both schemes. Importantly, MTGC performs better than HFedAvg with and without DP. We will include these new results in the revised version of our manuscript. [R1] Abadi, Martin, et al. Deep learning with differential privacy. ACM SIGSAC, 2016. [R2] Li, Bo, et al. An improved analysis of per-sample and per-update clipping in federated learning. ICLR, 2024

Reviewer fhg96/10 · confidence 3/52024-07-15

Summary

This paper proposes a novel algorithm called Multi-Timescale Gradient Correction (MTGC) for Hierarchical Federated Learning (HFL). The authors address the challenge of multi-timescale model drift in HFL systems, where data heterogeneity exists both within client groups and across groups. MTGC introduces coupled gradient correction terms to mitigate client model drift and group model drift. The authors provide theoretical convergence analysis for non-convex settings and demonstrate MTGC's effectiveness through experiments on various datasets and models.

Strengths

- The proposed MTGC algorithm is simple and easy to implement, introducing client-group and group-global correction terms. The paper clearly defines the problem it aims to solve. The motivation is clear and well-articulated, highlighting the gap in existing HFL algorithms. - The inclusion of theoretical analysis adds depth to the paper, providing a solid foundation for the proposed approach. The theoretical results show that MTGC achieves linear speedup in the number of local iterations, group aggregations, and clients. - The convergence bound of the proposed algorithm is immune to the extent of data heterogeneity, which is a significant strength.

Weaknesses

- There is a lack of comparisons and discussions on clustered federated learning [1,2], which share similar context to some extent and could provide valuable background for the readers. - The experiments primarily focus on image classification tasks. Including other types of tasks (e.g., natural language processing) could strengthen the generalizability of the results. There is a lack of experiments on various types of distribution shifts such as domain shift, label shift etc. Exploring different non-i.i.d. scenarios could provide more insights into the algorithm's robustness. - While the paper analyzes model drift theoretically, there is a lack of empirical and theoretical analysis on how model drift affects generalization performance, especially in relation to the number of hierarchical communication levels. The study also does not clearly demonstrate in which scenarios and to what extent increasing the number of hierarchical communication levels benefits federated learning performance. - The paper could also be strengthened by providing more insights into the practical implications of the theoretical results. [1] An Efficient Framework for Clustered Federated Learning. NeurIPS 2020. [2] Optimizing the Collaboration Structure in Cross-Silo Federated Learning. ICML 2023

Questions

Please refer to the weaknesses mentioned above.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations of the study are discussed, and no potential negative societal impacts of the work have been identified.

Area Chair XPt52024-08-12

Dear Reviewer fhg9, Could you please respond with how you think about the authors' response? Please at least indicate that you have read their responses. Thank you, Area chair

Reviewer r1S66/10 · confidence 3/52024-07-16

Summary

This paper presents a method to address multi-timescale model drift in hierarchical federated learning. Specifically, it introduces two control variables to correct intra-group client drift and group model drift. The paper establishes the convergence bound in a non-convex setup and demonstrates its stability against multi-level data heterogeneity. Overall, the paper is well-written and easy to follow.

Strengths

1. The writing is clear and easy to follow. 2. I have initially checked the proofs and did not find any issues so far.

Weaknesses

1. It would be beneficial to include comparisons of additional computational and communication costs of MTGC in the experimental section.

Questions

1. In Algorithm 1, consider using different color blocks to mark $x$, $z$, and $y$. This could help in better understanding the algorithm's process. 2. How about the runtime of the algorithm? Would it be possible to include comparisons of runtime with the baseline in the experiments? 3. In the experiments, does Eq.5 require the addition of regularization coefficients for correction terms $z$ and $y$? 4. Could you include some measurements of the additional communication costs in the experiments?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N.A.

Area Chair XPt52024-08-12

Dear Reviewer r1S6, Could you please respond with how you think about the authors' response? Please at least indicate that you have read their responses. Thank you, Area chair

Reviewer rcWg5/10 · confidence 4/52024-07-29

Summary

This paper introduces the usage of the gradient correction scheme to hierarchical federated learning. Specifically, the authors propose and analyze the multi-timescale gradient correction MTGC algorithm which is a direct generalization of SCAFFOLD to the framework where local clients aggregate their models on group server and group servers aggregate their models to a global server. The authors introduce control variables that correct i) client model drift and ii) group model drift that arises due to data heterogeneity. Theoretical convergence results are presented in the non-convex regime and experimental results showcase the ability of MTGC to address data heterogeneity both from clients and from group-servers.

Strengths

- The paper places itself correctly in the existing literature, is well-structured and easy to follow. - The problem of hierarchical FL is relevant and fairly interesting for the ML community. - The proposed algorithm is natural and easy to understand. - Both theoretical and numerical results are provided.

Weaknesses

- The main weakness of this work is its lack of novelty and technical contribution. The proposed MTGC is a straightforward extension of the well-known SCAFFOLD algorithm to a two-level hierarchical FL. Although the authors mention that the coupling of the two error correction variables is introducing new challenges I fail to see how this is the case. Indeed, the analysis of the correction variables appears to be largely the same as in SCAFFOLD and the authors to not make clear in the main body of their work what specific new challenges they faced and how they were circumvented. To summarize, although the paper is well written I do not see enough contribution to justify its acceptance to a top tier conference. - Minor typo: on line 110 in "However, their is" should be "However, their algorithm is".

Questions

See weaknesses section.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The authors adequately addressed the limitations of their work.

Authorsrebuttal2024-08-10

We really appreciate your reply and score update. We will include these discussions in the main body of the revised manuscript. Thanks again for the helpful suggestion.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC