Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer

Summary

This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. The authors propose the Fed-GraB to coordinate the global long-tailed distribution and the local learning strategy. Specifically, it consists of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients’ gradients in a closed-loop manner based on the feedback of global long-tailed prior derived from a Direct Prior Analyzer (DPA) module. Experiments are conducted on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.

Strengths

1. Investigating the federated long-tailed learning problem from the view of combining global and local sounds interesting. 2. The design of the proposed method is in line with the motivation of the paper, which seems technically feasible. 3. Experiments are conducted on many long-tailed benchmarks with various settings, which demonstrate the effectiveness of the proposed method.

Weaknesses

Although the technical contribution of the manuscript sounds qualified, the presentation of the manuscript is not clear enough. There are some confusing issues that I will explain in the questions section. Please resubmit a revised version. I will update my final score according to your revision. Meanwhile, too many hyperparameter settings will affect the generalizability of the proposed method.

Questions

1. What's the correlation between uj in Eq.3 and beta_j in Eq.4? And what's the correlation between beta_j and (beta_j^pos, beta_j^neg)? This is essential for understanding the re-weighting process of SGB. 2. In line 196, the author gives the definition of φ, but I could not find where the φ be applied. 3. In Eq.4, a random number r is adopted as a threshold for Pcj, what's the role of it in the SGB? 4. What's the definition of the expected target zj(t)? And why define the error feedback ej(t) as (gpos - gneg - zj(t))? 5. The EQLv2 strives to keep the cumulative positive and negative gradients to be equal for each category which seems to share similar ideas with the proposed method. Compared with EQLv2-FL, what are the novelty and advantages of this approach?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have declared the limitations in their paper.

Summary

This paper presents an approach, Fed-GraB, for addressing the challenges of Federated Long-Tailed Learning, an issue characterized by data heterogeneity and privacy concerns. The authors tested their method on several benchmark datasets, which significantly outperforms state-of-the-art baselines. The paper is mostly an experimental work.

Strengths

The proposed Fed-GraB model is based on an interesting technique called Self-adjusting Gradient Balancer to rebalance gradients. The experiments are comprehensive.

Weaknesses

The authors propose to tackle the challenges posed by data heterogeneity and privacy concerns in the FL setting. However, the paper does not sufficiently justify the uniqueness of their approach. A clearer and more compelling argument is needed on how the proposed Fed-GraB model uniquely and effectively addresses these challenges. The proposed model needs to be corroborated with theoretical analyses or at least insights, especially regarding the function and operation of the DPA modules. The paper often assumes that global class priors are available for re-balancing, which may not always be the case in real-world applications due to privacy constraints. More importantly, it's unclear how the proposed method would handle issues where local distributions are not long-tailed or present diverse long-tailed characteristics. A more comprehensive description of the interplay between the SGB and DPA modules, and the technical novelty there, would be helpful.

Questions

Please see the above comments

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

more technical analysis or discussions are needed

Summary

In this paper, a self-adjusting and closed-loop gradient re-balancing framework Fed-GraB was proposed and long-tailed learning tasks processing performance was improved.

Strengths

This paper presents a methodology named Fed-GraB, which incorporates a Self-adjusting Gradient Balancer (SGB) module. This module dynamically adjusts the weightage of gradients from individual clients in a closed-loop manner, guided by the feedback obtained from a Direct Prior Analyzer (DPA) module. By employing Fed-GraB, clients can effectively mitigate the challenges posed by disparate data distributions during the model training process. They achieve a global model that exhibits enhanced performance on underrepresented classes while maintaining the performance levels of the majority classes.

Weaknesses

1. Equation 4 lacks ending punctuation. 2. Figure 4(b) adds markers to make the curve more legible. 3. The definition of IFG is not clear. Please give a formula that says how to define IFG. 4. How to divide the dataset to get the Cifar-10/100-LT, there is no specific explanation. 5. The insight of the algorithm should be explained more clearly. 6. The baselines selected in this paper are not fully appropriate, because only one of them is designed for federated learning. The solutions to unbalanced data in FL are necessary to be compared.

Questions

Please see Weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

This paper considers the globally long-tailed distribution of data and its impact on federated learning (FL). To address this issue, the authors propose the Fed-GraB algorithm that re-balances gradients. The problem is timely and the proposed algorithm is novel. Overall, the paper is well-written, and simulations are extensive. However, the limitations of the proposed framework are not rigorously addressed via for instance mathematical or quantitative analysis. Furthermore, the applicability of the proposed framework to various applications and other state-of-the-art FL algorithms are questionable.

Strengths

The potential impact of the proposed framework would be significant, particularly on two emerging areas of research: fairness guarantee and rare-event detection. The proposed idea of re-balancing gradients is not entirely original, yet still sufficiently novel as it has yet been considered in FL for this purpose. Notably, the proposed idea does not incur significant costs in terms of computation and memory, which is a big plus for practical implementation. The literature of the considered problem and relevant frameworks is well reviewed, making the paper easily readable.

Weaknesses

Some potential weakness of the proposed framework has been proactively described, yet the claims are not very convincing. In particular, in the Privacy Discussions, it claims that "potential privacy issue exists in the general FL frameworks rather than specific to our proposed DPA method...As the privacy issue of FL framework is beyond the scope of this we briefly include the discussion in this subsection." This justification hardly responds to the issues when applying the proposed framework to other privacy-preserving FL frameworks, for instance, that applies quantization and noise injection under which the proposed framework's performance may be degraded. Furthermore, in Computational and Storage Cost of SGB, it claims that "the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap as well." This is not very convincing as it does not compare, for example, the resultant convergence speed and guarantee as well as FLOPS and bytes with the cases without the proposed solution.

Questions

1. According to (1), the proposed algorithm relies on classification and cross entropy loss, as opposed to the original FL that has no restriction on its task and loss function. Is the proposed algorithm applicable to 1) non-classification tasks as well as 2) non-cross entropy loss or cross-entropy with other regularizers? 2. There are various advanced FL algorithms that intentionally distort gradients via quantization, sampling, and noise injection for compression, privacy protection, and so forth. Is the proposed algorithm still applicable to these algorithms?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes