Communication-Efficient Federated Group Distributionally Robust Optimization

Summary

This work introduces three algorithms for communication-efficient Federated Group Distributionally Robust Optimization. The effectiveness of the proposed algorithms are verified through both theoretical and experimental results.

Strengths

1) This work studies an important problem of federated group distributionally robust optimization. 2) The theoretical results show the advantages of the proposed algorithms.

Weaknesses

1) This work proposes three algorithms, including FGDRO-CVaR, FGDRO-KL, and FGDRO-KL-Adam. There lacks a comparison between these algorithms. For example, what are the connections and differences between these algorithms? 2) The analysis for FGDRO-CVaR assumes the loss function to be rho-weakly convex, which is missing from the main context.

Questions

1) Missing reference: How about the comparison with this work [1]? [1] Communication-Efficient Distributionally Robust Decentralized Learning https://arxiv.org/pdf/2205.15614 2) What are the experimental setups for the number of clients and non-IID? 3) In experimental results (Tables 2 and 3), FGDRO-CVaR seems to have no advantages in both task; why do we need this algorithm? Besides, it is better to highlight the best-performance results in Tables 2 and 3. 4) Intuitively, using Adam optimizer can bring training speedup and is supposed to outperform other algorithms. But why do the results show that sometimes FGDRO-KL is better than FGDRO-KL-Adam? 5) In proof, is an assumption of bounded gradient needed? If I don't misunderstand, Line 550 indicates such an assumption. 6) If the loss function assumes to be convex, analyzing the optimal distance between the loss value the minimum loss should be better in Theorem 6.2.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness.

Summary

This paper addresses the challenge of reducing communication costs and sample complexity in Federated Group Distributionally Robust Optimization (FGDRO). The authors present the FGDRO-CVaR algorithm and the FGDRO-KL algorithm to address different constraints. Subsequently, they conduct extensive experiments across various real-world tasks, including NLP and CV tasks. The corresponding empirical results confirm the effectiveness of their proposed methods.

Strengths

1. The exploration of reducing communication costs for federated group DRO is a rarely-studied topic within the FL community. 2. The theoretical convergence analysis for the proposed algorithms is somewhat solid. 3. The authors conduct comprehensive experiments to validate the effectiveness of the devised algorithms.

Weaknesses

1. The contributions and novelties of this paper are unclear. It appears that the authors have directly combined existing federated adaptive algorithms with pre-existing federated group DRO methods in this paper.

Questions

1. The introduction's treatment of the concept of generalization appears incomplete. It is evident that there are two levels of generalization in Federated Learning, as delineated in [1] and [2]. 2. As highlighted in the aforementioned weaknesses, the authors should provide additional clarification regarding the contributions and novelty of this paper. Overall, it appears that the proposed method is primarily a direct combination of existing methods. 3. Similarly, the authors should delineate the challenges and innovations intrinsic to their theoretical analysis. Specifically, they should underscore the complexities involved in analyzing federated adaptive algorithms when applied in federated group DRO. 4. On line 154 of Page 4, what is the relationship between the "accurate estimate" and the "moving average" in the subsequent sentence? 5. Some minor points to address. The authors might consider offering more empirical results on convergence analysis in the experimental section. Additionally, they should further consider the statistical significance of these convergence analyses. [1] Hu X, Li S, Liu Y. Generalization bounds for federated learning: Fast rates, unparticipating clients and unbounded losses[C]//The Eleventh International Conference on Learning Representations. 2023. [2] Yuan H, Morningstar W, Ning L, et al. What do we mean by generalization in federated learning?[J]. arxiv preprint arxiv:2110.14216, 2021.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

The paper presents three methods for Federated Learning Group Distributionally Robust Optimization: (i) one tailored to reduce the CVaR which optimizes the top K-losses, (ii) another one tailored to tackle the KL divergence, and finally (iii) one that uses Adam locally. The paper is well written and the ideas are presented. To the best of my knowledge, the proofs are correct. My main concerns are regarding the relevance and importance of the subject, the lack of experiments, and the lack of empirical studies on communication efficiency.

Strengths

[S1] The paper is well-written, and the ideas are presented. [S2] The theoretical results are correct, to the best of my knowledge.

Weaknesses

[W1] The relevance of the subject is not entirely addressed. See [Q1] [W2] The experiment section is limited. In particular, the paper does not present any intuition on the problems they are solving. They do not consider the number of samples per server for example. I believe the authors should include a class imbalance problem [AN AGNOSTIC APPROACH TO FEDERATED LEARNING WITH CLASS IMBALANCE - Shen et al, ICLR 22]. [W3] Communication efficiency is not properly addressed by the authors. The authors show the number of communication rounds required, but they do not take into account how much is communicated. The authors claim that this method is more efficient in terms of communication, and they show it theoretically, but in the experiment section, there is no evidence of communication efficiency. I suggest the authors reveal the communication cost associated with each method, measured in the amount of data shared between servers. [W4] Privacy is an important subject of Federated Learning, but in this paper, there is no analysis of the privacy aspect. Can the authors elaborate on the privacy aspect of this work? [W5] Federated learning is a technique used to train on a set of machines. The idea is that the number of machines that participate is large. It appears to me that the largest number of servers is 17. This seems to me insufficient for a distributed learning problem.

Questions

[Q1] Why should be designed solutions that are distributional robust? And, at what cost? If we compare a method that simply maximizes/minimizes the FL problem, what is the overall loss? I believe the overall loss should be smaller, given that being distributionally robust is a particular case, and therefore, the unconstrained problem achieves a smaller minimum.

Rating

4

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

See weaknesses.

Summary

This paper aims to improve the efficiency of existing federated group distributionally robust optimization (FGDRO) when considering two specific types of regularization, condition value at risk and KL divergence. To address the first type of problem, the authors propose FGDRO-CVaR that reduces the sample complexity and communication costs simultaneously. For KL conditions, the proposed FGDRO-KL reduces the sample complexity while retaining the same communication costs. Moreover, the authors integrate the notion of Adam into FGDRO-KL, yielding FGDRO-KL-Adam and achieving better convergence speed.

Strengths

1. The paper is well-written, though some background information is missing. 2. The problem is well-motivated. The sample and communication efficiency is a pivotal problem in federated learning, though the benefits are not fully analyzed in the experiments. 3. The proposed method is grounded and improves over prior baselines.

Weaknesses

1. The background can be more thoroughly explained. The authors are encouraged to provide additional context to address the following questions, which will greatly enhance the paper's completeness. Why is federated group distributionally robust optimization (FGDRO) an important problem or technique? What are the sources of the additional communication costs? Why is it necessary to consider two different types of regularization? Are these types of regularization relevant to different applications? 2. My major concerns lie in the experiments and their settings. - **Data Splits.** While FGDRO's main advantage appears to be its ability to address non-IID optimization, the experimental setup concerning data splits lacks clarity. An analysis of the non-IID levels, such as those derived from different Dirichlet-distributed data splits with varying $\lambda$ values, is missing. Including more representative baselines, such as SCAFFOLD and FedProx, which are also designed for non-IID optimization, could further enhance the analyses. - **Performance.** The proposed method performs similarly to the baselines in most experiments. For example, in Tables 2 and 3, apart from the Adam variant, the proposed method is comparable to the baselines. This would be acceptable if the proposed method demonstrated improved efficiency; however, relevant analyses on this aspect are absent from the experiments. - **Communication or Sample Complexity Analysis.** An empirical analysis comparing complexity versus utility would be beneficial and highlight the advantages of the proposed method. For instance, the experiment in Figure 1 can be extended to a comparison among different baselines.

Questions

1. The datasets considered in this paper do not seem common in the existing literature. Could the authors report numbers on datasets like CIFAR-10/100 or EMNIST? Why did the authors choose the datasets in the paper?

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, the authors have discussed the limitations in the paper.

Thank you for the response

After reading other reviews and the authors' responses, I would keep my score. My major concern is that the bounded gradient assumption is somehow too strong.

Reponse

I appreciate the author's response. I choose to keep my score.