Library

Federated Learning with Manifold Regularization and Normalized Update Reaggregation

Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property for non-convex setting under partial client participation.Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

cs.LGPrivacy-Preserving Technologies in DataAdvanced Graph Neural NetworksComputer ScienceMathematics
View graphSource

Paper

References (46)

01ROLE: Rotated Lorentzian Graph Embedding Model for Asymmetric ProximityShanshan Feng, Lisi Chen, Kaiqi Zhao et al.2023 · IEEE Transactions on Knowledge and Data Engineering · 10 citations
02Efficient Federated Learning Via Local Adaptive Amended Optimizer With Linear SpeedupYan Sun, Li Shen, Hao Sun et al.2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 44 citations In Library
03Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth LandscapeYan Sun, Li Shen, Shixiang Chen et al.2023 · International Conference on Machine Learning · 78 citations In Library
04Enhance Local Consistency in Federated Learning: A Multi-Step Inertial Momentum ApproachYixi Liu, Yan Sun, Z. Ding et al.2023 · arXiv.org · 15 citations In Library
05FedExP: Speeding up Federated Averaging Via ExtrapolationDivyansh Jhunjhunwala, Shiqiang Wang, Gauri Joshi2023 · International Conference on Learning Representations · 98 citations In Library
06Generalized Federated Learning via Sharpness Aware MinimizationZhe Qu, Xingyu Li, Rui Duan et al.2022 · International Conference on Machine Learning · 234 citations In Library
07Federated Learning with Position-Aware NeuronsXin-Chun Li, Yi-Chu Xu, Shaoming Song et al.2022 · Computer Vision and Pattern Recognition · 42 citations In Library
08Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated LearningLin Zhang, Li Shen, Liang Ding et al.2022 · Computer Vision and Pattern Recognition · 430 citations In Library
09DENSE: Data-Free One-Shot Federated LearningJ Zhang, Chen Chen, Bo Li et al.2021 · Neural Information Processing Systems · 201 citations In Library
10Federated Learning Based on Dynamic RegularizationD. A. E. Acar, Yue Zhao, Ramon Matas Navarro et al.2021 · International Conference on Learning Representations · 1.2k citations In Library
11Fed2: Feature-Aligned Federated LearningFuxun Yu, Weishan Zhang, Zhuwei Qin et al.2021 · Knowledge Discovery and Data Mining · 95 citations In Library
12A Field Guide to Federated OptimizationJianyu Wang, Zachary B. Charles, Zheng Xu et al.2021 · arXiv.org · 488 citations In Library

Scroll for more · 34 remaining

Similar papers

Federated Learning under Partially Class-Disjoint Data via Manifold ReshapingZiqing Fan, Jiangchao Yao, Ruipeng Zhang et al.2024 · arXiv (Cornell University) · 5 citations In Library
Federated Learning Based on Model Discrepancy and Variance ReductionHao Zhang, Chenglin Li, Wenrui Dai et al.2025 · IEEE Transactions on Neural Networks and Learning Systems · 9 citations In Library
FedMR: Fedreated Learning via Model RecombinationMing Hu, Zhihao Yue, Zhiwei Ling et al.2022 · arXiv (Cornell University) In Library
ARFL: Adaptive and Robust Federated LearningMd. Palash Uddin, Yong Xiang, Borui Cai et al.2024 · IEEE Transactions on Mobile Computing · 14 citations In Library
FedAR: Addressing Client Unavailability in Federated Learning with Local Update Approximation and RectificationChutian Jiang, Hansong Zhou, Xiaonan Zhang et al.2024 · arXiv (Cornell University) · 2 citations In Library
Optimally Leveraging Stale Updates in Federated LearningHaoran Zhang2026 · ACM SIGMETRICS Performance Evaluation Review In Library
FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold PerspectiveZhekai Zhou, Shudong Liu, Zhaokun Zhou et al.2025 · arXiv.org · 0 citations In Library
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth LandscapeYan Sun, Li Shen, Shixiang Chen et al.2023 · arXiv (Cornell University) · 77 citations In Library
Stabilizing and Accelerating Federated Learning on Heterogeneous Data With Partial Client ParticipationHao Zhang, Chenglin Li, Wenrui Dai et al.2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 21 citations In Library
Is Aggregation the Only Choice? Federated Learning via Layer-wise Model RecombinationMing Hu, Zhihao Yue, Xie, Xiaofei et al.2023 · arXiv (Cornell University) · 36 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer EAwW5/10 · confidence 4/52023-07-04

Summary

The authors proposed a new algorithm for federated learning based on a Lorentzian regularization. The proposed algorithm achieves desired sub-linear convergence with linear speed-up. Numerical experiment shows the efficacy of the proposed algorithm over existing federated learning algorithms

Strengths

The proposed method is theoretically well-rounded with convergence guarantee for smooth non convex situation, which is the most commonly setting in the literature.

Weaknesses

(Please respond to the Questions section directly) The presentation of the paper is confusing. Also some technical details are not well-illustrated. The theory part of the paper is not clearly presented and seems to be abusing terminologies a lot.

Questions

Major: 1. The presentation of the paper is confusing: First, I cannot link the proposed Algorithm 1 with all the argument in Section 3.2. More specifically, how is the “mapped Lorentzian vectors” in (3) used in Algorithm 1? The authors claimed that the regularized problem (5) has a number of desired property but is Algorithm 1 solving this regularized problem? The argument in this section seems to be ending at nowhere; Second, I really cannot understand the usages of “hyperbolic” and “Lorentzian”. The hyperbolic space is defined as space with constant -1 curvature in mathematics, but here we are not dealing with curvatures at all. As for the Lorentzian regularizer defined in (4) and (5) (which again I don’t understand how it is used in Algorithm 1), why do we need Lorentz metric? What would happen if we just use Euclidean metric, or some Riemannian metric defined by a positive definite matrix? The entire section 3.2 is confusing. On the other hand, all the described operations in Section 3.2 are not well-reflected in Algorithm 1. For example, what’s $g_{i}^{t,k}$ in Algorithm 1? 2. As the authors claimed, the algorithm is built upon MoFedSAM[1]. How is the rate of convergence (in theory) of Algorithm 1 compared with existing works, especially MoFedSAM? If the theoretical rate is not improved, I would argue that this refined engineering over MoFedSAM might not be very interesting since (in my perspective) the only difference of Algorithm 1 with MoFedSAM is the $\Delta_i$ which serve as a local update corrector. 3. I cannot understand what point Figure 2 try to make. To me Figure 2(b) is a rotation operation, not projection, also I don’t understand how is the operation used in Algorithm 1. Minor: 1. In the abstract, the authors include a formula. I think it’s better to either explain the meaning of each of the variable in the formula, or just use words instead; 2. What does the name “FedMRUR” exactly stand for? 3. Typos: Line 227, Assumption 1, missed one $\nabla$; Line 263 “Dirichlet… from {0.30.6}” missed a comma? References: [1] Qu, Zhe, et al. "Generalized federated learning via sharpness aware minimization." International Conference on Machine Learning. PMLR, 2022.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

The limitation is well stated in weakness and question sections. I’m not aware of any potential negative social impact of this work.

Reviewer jAXP5/10 · confidence 3/52023-07-05

Summary

The authors propose a novel Federated Learning Algorithm, FedMRUR that uses hyperbolic graph diffusion to reduce the effect of data heterogeneity and thereby model inconsistencies. The authors also propose a normalized aggregation scheme to achieve faster convergence. The algorithm FedMRUR achieves state-of-the-art performance on standard datasets.

Strengths

In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM.

Weaknesses

Finally, while convergence speed is quicker for FedMRUR, Table 3 in the Supplemental Materials actually has FedCM to have the quickest total time to achieve 60% test accuracy on CIFAR-100. Do the authors have comments or suggestions on when to use FedMRUR vs FedCM given the disparity between convergence speed and clock times?

Questions

1. The authors should clarify notation in Figure 1 and Algorithm 1. In Figure 1, the global parameter is subscripted, w_0, but superscripted in the algorithm. 2. The algorithm can be hard to follow as it: a. Did not specify the global parameter \eta_g in the parameters b. Did not specify the gradients, g_{i}^{t,k}, though that is easily induced c. More importantly did not specify how \tilde{g}_{i}^{t,k} is derived. This is mentioned later in line 213, but would make the presentation clearer by linking it directly to the equations d. They can also specify the normalized aggregation more clearly, and tie to the algorithm 1. 3. The authors should discuss normalized aggregation in more depth, as it seems to contribute more than hyperbolic graph fusion (Table 3). In fact, it would be interesting to understand the effect of this operator in other baselines. 4. In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM. First, it would be useful to have test accuracy and test accuracy std to enable better comparison between comparably performing models. Second, how was Table 1 derived, as the accuracy numbers are quite different and higher than the comparable Table 5 in the MoFedSAM paper for CIFAR-100 with Dir(). 5. Similarly, the convergence table 2 seem different for MoFedSAM and FedAvg vs Table 5 in the MoFedSAM paper. Is this a setup difference?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have addressed some limitations of their method

Reviewer aqu47/10 · confidence 3/52023-07-06

Summary

This paper studies the problem of model inconsistency across clients in federated learning (FL). The authors propose a method called FedMRUR, which uses a hyperbolic graph manifold regularization term and a normalized update aggregation scheme to alleviate the issues introduced by model inconsistency. Compared with the previous works, the proposed FedMRUR can reflect the model bias better in a low-dimensional subspace and mitigate the norm reduction of global updates caused by model inconsistency. The authors prove the convergence of FedMRUR for nonconvex objectives under partial client participation. They also run experiments to show that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Strengths

1. The work is written clearly and intuitively. 2. The considered problem is quite meaningful as the model inconsistency severely impairs the performance of FL and building an algorithm to solve this issue is a promising direction to improve FL. 3. The theoretical result is rigorous for non-convex settings under partial client participation. 4. There are experiments to support the theory as well as to show that the algorithm will be useful in practice.

Weaknesses

See questions

Questions

1. It is not clear what is the “manifold structure of their representations”. 2. What does it mean by “norm reduction” and why it is important? 3. For the non-i.i.d pathological-n setting on TinyImageNet, line 264 conflicts with Figure 3(b). 4. Does the setting of the parameter $\beta$ for hyperbolic graph manifold regularization have an impact on the performance of the algorithm? 5. In the ablation study, the author should discuss the hyperparameters sensitivity of $\gamma$ in a wider range.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the Questions.

Reviewer HiX97/10 · confidence 3/52023-07-06

Summary

The authors found the existing vanilla distillation in FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Moreover, the authors argue previous works may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space’s limitation in meaningful geometric representations. To resolve the above issues, they propose the FedMRUR algorithm for FL. By adopting a hyperbolic graph manifold regularizer and aggregating the client updates norms as the global update norm, the FedMRUR achieves a new state-of-the-art (SOTA) accuracy with less communication.

Strengths

originality: Exploiting the manifold structures of the representations can reflect the model bias better than the parameters (or gradients) method, to significantly reduce the model inconsistency. Aggregating the client updates norms as the global update norm can mitigate the norm reduction caused by model inconsistency. quality: The paper is well written, with detailed experiments and ablation studies with other methods and the proposed variants. clarity: The paper consists of an illustration of the workflow and text explanations of the proposed FedMRUR algorithm. significance: The paper solves the problem of the model inconsistency across the clients in FL, reduces the model bias more effectively than its baseline, mitigates the norm reduction caused by model inconsistency, and improves the test performance.

Weaknesses

- There are some unclear points and confusing notations. - More experiments can be conducted.

Questions

1) Some notations are confusing. For example, in line 14 of Algorithm 1, what does the notation “$\delta_i^{t}$” mean? 2) In line 263, what should the setting of the Dirichlet coefficient be? 3) The authors should discuss the effect of local interval $K$ on the performance of the algorithm.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please find the weaknesses and limitations.

Reviewer mqsa5/10 · confidence 3/52023-07-26

Summary

This paper proposes a federated learning framework called FedMRUR to deal with the model inconsistency caused by the local data heterogeneity across clients and insufficient geometric representation ability. To do this, it adopts a hyperbolic graph manifold regularizer to ensure that the representations obtained by the local and global models are close in a low-dimensional subspace. Then it aggregates the client updates norms as the global update norm to mitigate the norm reduction introduced by the near-orthogonality of client updates. A linear speedup property is theoretically proved for the proposed algorithm.

Strengths

+ The motivation to deal with the model inconsistency and insufficient representation ability is clear. + It is interesting to adopt the hyperbolic graph fusion scheme in federated learning. + The theoretical guarantee is provided for the convergence and especially the proposed method exhibits a linear speedup. + The experimental results seem promising.

Weaknesses

I'm not very familiar with federated learning. Here are some major concerns: - There are many ways to exploit the manifold structure. What's the advantage of adopting the graph fusion scheme, especially in the hyperbolic space? If the hyperbolic space is a good choice, then why not let the whole learning process work in the hyperbolic space? - Figure 2 is less informative since the readers cannot understand how the normalized aggregation works and what's the difference between these two operations in terms of their workflow. - It seems that the widely used CIFAR-10 benchmark is not involved for evaluation. - The experiments to validate the linear speedup property are missing.

Questions

Please see the above part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

NA

Reviewer EAwW2023-08-14

I thank the authors for responses to my questions. May I know where I can find the revised version of the paper? For example I cannot find Theorem 9 mentioned in the rebuttal. I still have some concerns with this work: The first issue is that adding the extra hyperbolic regularization may increase the testing accuracy, but the justification in the author's rebuttal "In FL, clients and the server logically also have a tree-like hierarchical topological relationship, so adopting the Lorentz metric of hyperbolic space makes use of the hierarchical information in the datasets and the hierarchical relationship between the server and clients, which are helpful to group data samples further bring prediction gains" seems too strong and general to be just supported by current evidences. However this is still not the biggest issue that I'm concerned, since the observation may be a good starting point for incorporating geometric information in FL. The authors could give a comprehensive study on the effectiveness of different regularization terms as a future work to support their broad claim. My biggest concern is that I'm still not sure if the proposed algorithm is theoretically better than MoFedSAM[1]. As in the rebuttal "The main dominated term of convergence rate is ..., the convergence rate of FedMRUR is faster than MoFedSAM". Can the authors illustrate why the rate $(F^0-F^*)/(c\alpha\eta_g \sum_{t=1}^{T}d_t) + \Phi$ is better than $(F^0-F^*)/(c\alpha\eta_g T)$ in [1], as the authors wrote in the rebuttal? I'm still very concerned and confused here. [1] Qu, Zhe, et al. "Generalized federated learning via sharpness aware minimization." International Conference on Machine Learning. PMLR, 2022.

Authorsrebuttal2023-08-15

**-Q1: May I know where I can find the revised version of the paper? Where is Theorem 9.** -A1: We are sorry about that you can't find the revised paper. Due to the limitations of the rebuttal system, we can't upload the revised paper. We re-analyze the convergence rate of the algorithm to better illustrate the effect of **normalized aggregation**. Theorem 9 is the revised version of Theorem 8 in the supplementary file. Compared with Theorem 8, under the same condition, Theorem 9 gives a tighter upper bound on the convergence rate, which is $$\frac{1}{\sum_{t=1}^{T} d_t} \sum_{t=1}^{T} \mathbb{E} \left\lVert \nabla F(w^t) \right\rVert^2 d_t \le \frac{F^0 - F^*}{C \alpha \eta_g \sum_{t=1}^{T} d_t} + \Phi.$$ In A2(the reply to Q2), we present our revision of the proof of Theorem 9. **-Q2: Can the authors illustrate why the rate $\frac{F^0 - F}{C \alpha \eta_g \sum_{t=1}^{T} d_t} + \Phi$ is better than $\frac{F^0 - F}{C \alpha \eta_g T}$ in MoFedSAM paper?** ($F=F^*$) -A2: Sorry for the typo about the convergence rate of MoFedSAM. The convergence rate should be $\frac{F^0 - F^*}{C \alpha \eta_g T} + \Phi$ (Theorem D.4 in MoFedSAM paper). Due to the limit of the reply, we present the revised part of the proof and illustrate why FedMRUR is better in terms of convergence rate. At the beginning of the proof of Lemma 7, we define $d_t= {\sum_{i \in S_t}\Vert \triangle_i^t \Vert }/{\Vert \sum_{i \in S_t} \triangle_i^t \Vert} \ge 1$, which is obtained by **normalized aggregation**, and introduce $\varepsilon_{\delta}$ from Lemma B.1 in MoFedSAM paper. For the 2nd and 3rd term in the R.H.S of (6), we multiply them by $d_t$. Consequently, we multiply the entire R.H.S of (7) by $d_t$. Finally, we rewrite (8) and draw a new conclusion for Lemma 7: $$\mathbb{E}_t [ F(\tilde{w}^{t+1})] \le F(\tilde{w}^t) - K \eta_g \eta_l d_t (\frac{1}{2} - 20K^2L^2\eta_l^2)\left\lVert \nabla F(\tilde{w}^t) \right\rVert^2 + K\eta_g\eta_l\left( 6K^2\eta_l^2\alpha^4\rho^2 + 5K^2\eta_l\alpha^4\rho^2\sigma^2 + 20K^3\eta_l^3\alpha^2\sigma_g^2 + 16K^3\eta_l^4\alpha^6\rho^2 + \frac{\eta_g\eta_l\alpha^3\rho^2}{N} \sigma_l^2 \right)$$ Summing this new inequality over $t$ and multiplying both sides by $\frac{1}{C \alpha \eta_g \sum_{t=1}^T {d_t}}$, we have $$\frac{1}{\sum_{t=1}^{T} d_t} \sum_{t=1}^{T} \mathbb{E} \left\lVert \nabla F(w^t) \right\rVert^2 d_t \le \frac{F^0 - F^*}{C \alpha \eta_g \sum_{t=1}^{T} d_t} + \Phi \le \frac{F^0 - F^*}{C \alpha \eta_g T} + \Phi,$$ where $\Phi$ is the same as the $\Phi$ in MoFedSAM. The first term is tighter than the one in MoFedSAM since $d_t \ge 1$. There, we get the conclusion of Theorem 9 (revised version of Theorem 8). With a proper choice of $\eta_g$, $\eta_l$ and $\rho$, the convergence rate can be rewritten as $O (\frac{1}{\sqrt{SKT}}) + O \left(\frac{\sqrt{K}}{{ST}} \right) + O \left( \frac{1}{\sqrt{K}T} \right)$, where the main dominated term has the same order with the convergence rate of MoFedSAM (Theorem 4.1 in MoFedSAM paper). Our derived bound agrees with MoFedSAM in order ($O (\frac{1}{\sqrt{SKT}})$), and our convergence analysis illustrates the effect of **Normalized Aggregation** on constants. **-Q3: The first issue is the justification in the rebuttal about hyperbolic space.** -A3: In FL, the clients train local models with their own data and the server merges these models into the shared global model. The shared global model should contain as much common information from each client as possible. The global model can be seen as the tree's root, and the local model is viewed as the leaf. In addition, from the perspective of the model structure, deep neural networks can be modeled as graphs[1,2] due to their multi-layered structures containing hierarchical information. Since its volume increases exponentially with its radius, the hyperbolic space can represent hierarchies with minimal distortion[3] and reduce the number of embedding dimensions[4]. Therefore, embedding deep neural networks makes model fusion more effective and improves performance. We also conduct experiments embedding deep neural networks in different geometric spaces using multiple seeds. The results are as follows: | Space | Euclidean | Hyperbolic | | :-------: | :-------: | :--------: | | Test Acc. |67.46(0.32)| 68.99(0.35)| From these results, we can see that embedding model in hyperbolic space improves the test accuracy of the aggregated global model. Reference: 1. Singh et.al, "Model Fusion via Optimal Transport", NeurIPS 2020 2. Liu et.al, "Deep Neural Network Fusion via Graph Matching with Applications to Model Ensemble and Federated Learning", ICML 2022 3. M. Gromov, “Hyperbolic groups”, Essays in Group Theory, 1987 4. Peng et.al, "Hyperbolic Deep Neural Networks: A Survey", IEEE Trans. PAMI 2022 Thank you very much for your valuable comments, which help us to improve our work. If you have any further questions about our submission and rebuttal, please let us know.

Reviewer EAwW2023-08-15

I thank the authors for further clarifications. Let me further illustrate my points: 1. The $\sum_{t=1}^{T}d_t \geq T$ is indeed an interesting observation, however this also indicates that the rate of convergence in theory is the same as MoFedSAM algorithm in the worst case. Also one point I forgot to mention is that the function $F$ is the regularized function, not the original function in the FL setting in equation (1), whereas the $F$ in MoFedSAM is the original FL problem (they didn't do any regularization), therefore the convergence theory in this paper seems flawed to me. 2. Again I still believe that the testing accuracy comparison between Euclidean and Hyperbolic regularizations not adequate to support the claim of the authors. The hyperbolic regularization is incorporated with a neural network which may lead to some complication in parameter tuning and algorithm design for general applications. I think I could better appreciate the beauty of the regularization if there isn't a complicated neural network embedding. In short, I think the direction of adding regularization to improve convergence for FL is an interesting approach, yet I think this work is entangled with too many blocks where the effect of each of the block seems unclear to me. The convergence theory doesn't show a clear improvement over the SOTA. To AC: Unless there are more decisive arguments from the authors or other reviewers, I decide to keep my score and evaluation. I didn't see any reply from other reviewers, and I will keep an eye.

Authorsrebuttal2023-08-15

Further clarification on the theoretical results and hyperbolic regularization.

Dear reviewer, thanks for your quick reply and valuable comments, which help us a lot to improve our submission. Thanks for recognizing our theoretical contribution on improving the constant of the optimal bound and novel hyperbolic regularization for model aggregation. **-Q1:The $\sum_{t=1}^{T} d_t \ge T$ is indeed an interesting observation, however this also indicates that the rate of convergence in theory is the same as MoFedSAM algorithm in the worst case. Also one point I forgot to mention is that the function $F$ is the regularized function, not the original function in the FL setting in equation (1), whereas the $F$ in MoFedSAM is the original FL problem (they didn't do any regularization), therefore the convergence theory in this paper seems flawed to me.** -A1: Firstly, in the worst case, the convergence rates of the current first-order optimizers are all optimal including FedAgv, FedProx, MoFedSAM, etc., and the only thing that has changed for all of them is the difference in some constant terms. Theoretically, we can see that there is some improvement in the constant terms for our proposed method, as the reviewer said. In fact, the convergence rates of FedSAM and MoFedSAM, and FedAvg are also in the same order, because all of them are optimal, and it is not possible to have an improvement in the order unless some stronger assumptions are adopted. Secondly, the regular term is due to hyperbolic regularization, not due to normalized aggregation, and we have theoretically verified the role of normalized aggregation to some extent, even if normalized aggregation is used in MoFedSAM. This contribution is also highly nontrivial and demonstrates the benefit of the normalized aggregation technique. Thirdly, we also record the values of $\frac{\sum_{t=1}^T d_t}{T}$ over different $T$ during FL training on the CIFAR100 dataset with Dirichlet 0.3 and the results are as follows: | T | 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000 | 1100 | 1200 | 1300 | 1400 | 1500 | 1600 | |:--: | :--: | :-: | :--:| :--:|:--: | :-: | :--:| :--:| :--:| :--: | :--: | :--: | :--: | :--: | :--: | :--: | |value| 1.110|1.195|1.313|1.406|1.470|1.514|1.545|1.568|1.586| 1.601|1.613 |1.623 |1.632 |1.639 |1.645 |1.650 | From this result, we can find that $\forall T$, $\frac{\sum_{t=1}^T d_t}{t} > 1$ and $\frac{\sum_{t=1}^T d_t}{t}$ is increasing as $T$ increasing, which also empirically verifies the improved constant of our proposed approach. **-Q2: Again I still believe that the testing accuracy comparison between Euclidean and Hyperbolic regularizations not adequate to support the claim of the authors. The hyperbolic regularization is incorporated with a neural network which may lead to some complication in parameter tuning and algorithm design for general applications. I think I could better appreciate the beauty of the regularization if there isn't a complicated neural network embedding.** -A2: First of all, we also agree with the reviewer's opinion on the philosophy of Occam's Razor in machine learning, i.e., the simpler it is, the closer it tends to be to the truth. However, it is also valuable to design appropriate algorithms according to the structure of the problem. Although hyperbolic regularization involves some complicated mathematics, it is not a disadvantage, but an advantage in model aggregation for FL with complex deep neural networks, because the neural network itself can be considered as a very high-dimensional graph, and it is difficult for the naive model aggregation to achieve neural network embedding well; while hyperbolic regularization used in our work can solve this problem well. Our experiments and ablation study can also illustrate its benefits. At last, thank you for your kind effort in helping us improve our submission. If you have further comments, please let us know.

Authorsrebuttal2023-08-18

Dear Reviewer EAwW: We really appreciate your suggestions and active discussions on our paper which largely help us improve our work. All these valuable discussions will be incorporated into our revision. In the rolling discussion, we have tried our maximum effort to address your concerns, but it has been a while since our last comments. Currently, we are not sure whether our response has solved your concerns and we are on the same page with you now. If you have further suggestions, please let us know. Thanks again for your valuable comments and active discussions. Best, Authors.

Authorsrebuttal2023-08-20

Dear Reviewer EAwW, As the discussion phase is drawing to a close, we kindly ask you whether our further explanations have addressed your concerns. If you have any further concerns, we are happy to discuss them with you. Thanks again for your valuable comments and active discussions. Sincerely, Authors.

Reviewer EAwW2023-08-20

I thank the authors for more empirical evidences. Summary of my mind right now: 1. The normalized aggregation brings interesting results both theoretically and empirically, which is a good point; 2. The hyperbolic regularization still seems doubtful to me. The authors provides some comparisons over the test accuracy on Cifar-100 with the Euclidean regularization but I'm still deeply concerned with this point; 3. I appreciate the authors effort to address my concerns with extensive numerical evidences, and the authors indeed addressed my concern with the normalized aggregation. I see that the other reviewer increased the point after the authors provided some comparison with FedCM and other algorithms; 4. The original paper, as I look back, was indeed not well-organized. It's a pity that I couldn't see the updated version. I have a mixed sentiment toward this work, that some part of it seems interesting but some part of it is doubtful, and the whole work is a mixture of different ideas. I'd say I will give a weak accept after careful consideration. To AC: I'll increase my point but please keep in mind that I still have some concerns regarding both the theory and the general layout of the work when you make the decision.

Authorsrebuttal2023-08-21

Thanks for your valuable comments and suggestions, which largely help us improve our submission. In the next version, we will make the arguments more clear. We really appreciate your effort.

Reviewer jAXP2023-08-16

I thank the authors for their response, which clarified questions 2-5. However, I am not convinced by their argument for run times in their response to Q1. I understand that communication rounds present a bottleneck, and is a standard for many FL studies, while other operations can be performed locally and asynchronously. However, does the high computation cost of SAM and Hyperbolic Graph Fusion, as stated by the authors, limit the application of FedMRUR to a more narrow range of local devices thereby limiting FedMRUR?

Authorsrebuttal2023-08-16

FedMRUR is more efficient with respect to both the communication round and wall-clock time when high-performance models are required.

Dear reviewer, thanks for your quick reply and valuable comments, which help us a lot to improve our submission. **-Q: However, does the high computation cost of SAM and Hyperbolic Graph Fusion, as stated by the authors, limit the application of FedMRUR to a more narrow range of local devices thereby limiting FedMRUR?** -A: To answer this question, here we report the communication rounds, sampled gradient, and wall-clock time used to achieve different test accuracy on CIFAR100 with FedMRUR, FedCM, and MoFedSAM. | Rounds vs. Acc. | 40% | 45% | 50% | 55% | 58% | 60% | 61% | 62% | 63% | | :--------------:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | FedCM | 160 | 200 | 249 | 307 | 390 | 464 | 543 | 684 | 994 | | MoFedCM | 177 | 217 | 258 | 315 | 364 | 403 | 447 | 492 | 573 | | FedMRUR | 198 | 219 | 253 | 288 | 318 | 352 | 372 | 411 | 438 | | Sampled Gradient (x1000) vs. Acc. | 40% | 45% | 50% | 55% | 58% | 60% | 61% | 62% | 63% | | :-------------------------:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | FedCM | 84 | 100 |124.5|153.5| 195 | 232 |271.5| 342 | 497 | | MoFedCM | 177 | 217 | 258 | 315 | 364 | 403 | 447 | 492 | 573 | | FedMRUR | 198 | 219 | 253 | 288 | 318 | 352 | 372 | 411 | 438 | | Wall-clock time(s) vs. Acc.| 40% | 45% | 50% | 55% | 58% | 60% | 61% | 62% | 63% | | :-------------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:--------:| | FedCM | 2205.84 | 2626.00 | 3269.37 | 4030.91 | 5120.70 | 6092.32 | 7129.59 | 8980.92 | 13051.22 | | MoFedCM | 3571.86 | 4379.06 | 5206.44 | 6356.70 | 7345.52 | 8132.54 | 9020.46 | 9928.56 | 11563.14 | | FedMRUR | 4369.86 | 4833.33 | 5583.71 | 6356.16 | 7018.26 | 7768.64 | 8210.04 | 9070.77 | 9666.66 | Due to the double calculation of the gradients via SAM, MoFedSAM, and FedMRUR take more time in a single round. However, the communication rounds required are much less than FedCM. From these tables, we can find that FedMRUR requires the least wall-clock time and sampled gradients to achieve high test accuracy due to the fastest convergence of FedMRUR under the same hardware conditions. Considering the total wall-clock time costs, the acceleration ratio of FedMRUR achieves $1.20 \times$ compared with MoFedSAM ($1.35 \times$ compared with FedCM) at the final. Therefore, we can conclude that the FedMRUR is more efficient with respect to the communication round, gradient calculation, and wall-clock time when high-performance models are required. At last, thank you very much for your valuable comments, which help us to improve our work. If you have any further questions about our submission and rebuttal, please let us know.

Reviewer jAXP2023-08-16

This table is interesting as it suggests that FedMRUR eventually reaches higher test accuracy (63%) quicker than FedCM and MoFedCM, but that is only 1 data point, and it might be difficult to have more test accuracy points higher to validate the point further. I agree the acceleration ratio appears lower.

Authorsrebuttal2023-08-16

Quick reply.

Dear reviewer, thanks for your reply. Below, we first give a quick reply to your comments. Once we finish the full table with more target accuracy data points, we will post it as soon as possible. **-Q: This table is interesting as it suggests that FedMRUR eventually reaches higher test accuracy (63%) quicker than FedCM and MoFedCM, but that is only 1 data point, and it might be difficult to have more test accuracy points higher to validate the point further. I agree the acceleration ratio appears lower.** -A: Since during the rolling discussion stage, we can not upload the figure version of the above three tables. In fact, we can further increase the target accuracy data point to greater than 68%. In this setting, both the FedCM and MoFedSAM are hard to achieve this target accuracy (with more than 1500 communication rounds). The acceleration ratio will be more clear (larger than $2\times$ or even more). Once we finish the full table, we will print it to show the large acceleration ratio. From the theoretical perspective, all the FedCM, MoFedSAM, and FedMRUR are first-order optimizers with a sublinear convergence rate. Thus, they struggle to increase the accuracy at the late stage. Please check Table 1 and Figure 3 in the submission. Both FedCM and MoFedSAM are almost stagnated to achieve 65% accuracy. While thanks to the introduced normalization and hyperbolic regularization techniques, FedMRUR can achieve a higher accuracy, which results in a high acceleration ratio in the late stage. **In summary, when higher accuracy is required, the acceleration ratio of FedMRUR over FedCM and MoFedSAM will be much larger.** Hope this quick reply can partially solve your concern. We will also incorporate the full table in our revision.

Authorsrebuttal2023-08-17

Larger acceleration ratio can be achieved by FedMRUR

Dear reviewer, thanks for your reply. Here we present the full table with more test accuracy points. The table recording gradient calculations, communication rounds, and wall-clock time to achieve various accuracy ($\ge$ 62.5%) on CIFAR100 with FedMRUR, FedCM, and MoFedSAM are as follows: | Rounds \ Acc. |62.5%|62.7%|62.9%|63.1%|63.3%|63.5%|63.7%|63.9%|64.1%|64.3%|64.5%|64.7%|64.9%|65.1%|65.3%|65.5%|65.9%|66.3%|66.7%|67.1%| | :-------------:|:---:|:---:|:---:| :-: |:---:|:---:|:---:|:---:|----:|:---:|:---:|:---:|:---:| :-: | :--:|:---:| :--:|:---:|:---:|:---:| | FedCM | 749 | 873 | 933 | 997 | 999 | 1166| 1395| 1395| 1396| 1561| 1727| --- | --- | --- | ----| ----| ----| ----| ----| ----| | MoFedCM | 525 | 548 | 573 | 585 | 586 | 604| 606| 645| 662| 747| 855| 1080| 1194| 1295| 1421| 1512| 1561| 1623| 1662| ----| | FedMRUR | 421 | 428 | 438 | 441 | 450 | 466| 500| 502| 502| 544| 544| 559| 591| 594| 640| 652| 674| 704| 728| 786| | Gradients(x1000)\Acc.|62.5%|62.7%|62.9%|63.1%|63.3%|63.5%|63.7%|63.9%|64.1%|64.3%|64.5%|64.7%|64.9%|65.1%|65.3%|65.5%|65.9%|66.3%|66.7%|67.1%| | :-------------------:|:---:|:---:|:---:| :-: |:---:|:---:|:---:|:---:|----:|:---:|:---:|:---:|:---:| :-: | :--:|:---:|:---:|:---:|:---:|:---:| | FedCM |374.5|436.5|466.5|498.5|499.5| 583|697.5|697.5| 698|780.5|863.5| --- | --- | --- | ----| ----| ----| ----| ----| ----| | MoFedCM | 525 | 548 | 573 | 585 | 586 | 604| 606| 645| 662| 747| 855| 1080| 1194| 1295| 1421| 1512| 1561| 1623| 1662| ----| | FedMRUR | 421 | 428 | 438 | 441 | 450 | 466| 500| 502| 502| 544| 544| 559| 591| 594| 640| 652| 674| 704| 719| 756| |time(s) vs. Acc.| 62.5% | 62.7% | 62.9% | 63.1% | 63.3% | 63.5% | 63.7% | 63.9% | 64.1% | 64.3% | 64.5% | 64.7% | 64.9% | 65.1% | 65.3% | 65.5% | 65.9% | 66.3% | 66.7% |67.1%| |:--------------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:| :----: | :----: | :-----:|:------:|:------:|:------:|:------:|:---:| | FedCM | 9821.40|11462.49|12250.29|13090.61|13116.87|15309.58|18316.35|18316.35|18329.48|20487.56|22664.28| ---- | ----- | ------ | -------| -------| -------| -------| -------| ----| | MoFedCM |10594.50|11058.64|11563.14|11805.30|11825.48|12188.72|12229.08|13016.10|13359.16|15063.29|17235.73|21782.48|24083.88|26132.94|28657.26|30512.16|31500.98|32752.14|33539.16|--| | FedMRUR | 9291.47| 9445.96| 9666.66| 9717.53| 9921.31|10284.62|11035.00|11079.14|11079.14|12001.36|12001.36|12327.16|13024.54|13044.84|14124.25|14389.64|14949.32|15614.72|15947.04|16768.08| From this table, we can find that FedMRUR takes the least wall-clock time, and the acceleration ratio increases from $1.05 \times$ (62.5% accuracy) to $1.88 \times$ (64.5% accuracy) compared with FedCM and FedCM can't achieve 64.7% accuracy. Compared with MoFedSAM, the acceleration ratio increases from $1.05 \times$ (60% accuracy) to $2.10 \times$ (66.7% accuracy) and MoFedSAM can't achieve 67.1% accuracy. We can conclude that because of the normalized aggregation and hyperbolic regularization techniques, FedMRUR can accelerate the training time and achieve high test accuracy. **These tables further verify that when higher accuracy is required, the acceleration ratio of FedMRUR over FedCM and MoFedSAM will be much larger. We hope this verification can solve your concerns completely** At last, thank you very much for your valuable comments, which help us to improve our work. If you have any further questions about our submission and rebuttal, please let us know.

Authorsrebuttal2023-08-18

Dear Reviewer HiX9: We really appreciate your constructive opinions that helped us improve this paper. All the discussions will be incorporated into our revision. If there are any concerns unresolved, we would be glad to have further discussions. Thanks again for your time. Best, Authors.

Authorsrebuttal2023-08-18

Dear Reviewer aqu4: We really appreciate your constructive opinions that helped us improve this paper. All the discussions will be incorporated into our revision. If there are any concerns unresolved, please let us know and we are ready to have further discussions with you. Thanks again for your time. Best, Authors.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC