Coordinating Distributed Example Orders for Provably Accelerated Training

Recent research on online Gradient Balancing (GraB) has revealed that there exist permutation-based example orderings for SGD that are guaranteed to outperform random reshuffling (RR). Whereas RR arbitrarily permutes training examples, GraB leverages stale gradients from prior epochs to order examples -- achieving a provably faster convergence rate than RR. However, GraB is limited by design: while it demonstrates an impressive ability to scale-up training on centralized data, it does not naturally extend to modern distributed ML workloads. We therefore propose Coordinated Distributed GraB (CD-GraB), which uses insights from prior work on kernel thinning to translate the benefits of provably faster permutation-based example ordering to distributed settings. With negligible overhead, CD-GraB exhibits a linear speedup in convergence rate over centralized GraB and outperforms distributed RR on a variety of benchmark tasks.

Paper

References (54)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer T8gT6/10 · confidence 2/52023-06-24

Summary

This paper is devoted to the development of the Gradient Balance for the centralized distributed setting. The proof is highly non-trivial because the extension to distributed ML is not natural. To this end, the authors use kernel thinning to translate the benefits of provably faster permutation-based example ordering to distributed settings.

Strengths

1. This paper proposes the online Coordinated Distributed Gradient Balancing (CD-GraB) algorithm enabling provably accelerated training in the parallel setting. The new algorithm leverages insights from kernel thinning to elevate the herding framework of centralized GraB (GraB) to the parallel setting. 2. They prove that the convergence rate for CD-GraB exhibits a linear speedup over GraB, which is non-trivial. 3. Empirical results are presented to demonstrate the efficiency of the proposed algorithm.

Weaknesses

1. The presentation may need to be improved. For example, if we turn to Table 1 (Algorithm 1), the algorithm is not self-explanatory because some notation are not defined. 2. The paper uses a different smoothness assumption. The authors seem to want to hide the degradation from the dimension $d$. 3. The technical difficulty of convergence analysis can be presented as roadmap for the reader.

Questions

What is the main technical difficulty of convergence analysis for distributed GraB. Minor: line 32: "Gradiant Balance algorithm" should be "Gradient Balance algorithm"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Reviewer xnQq5/10 · confidence 4/52023-07-06

Summary

Propose an online method, CD-GraB for shuffling data samples during distributed model training (parameter-server setting) in a fashion that accelerates convergence in both theory and practice. Theoretical assumptions are reasonable and quite standard in the literature. Empirically, this work mostly builds on the GraB method, which was described for centralized settings, and extended to the distributed setting in this work. The difficulty in extending the algorithm to the distributed setting is that * the considered problem setting only allows workers to produce permutations of their local data samples * large step sizes results in relatively large disparities across gradients at different workers

Strengths

**Quality** The work appears to be of a high quality. The proposed method and proofs are non-trivial, and the problem addressed by this work, building on the GraB method, requires careful thought. **Clarity** Diagrams are polished, notation is reasonable. The proposed method is non-trivial, and the authors have done a reasonable job at explaining the procedure, with additional illuminating diagrams in the appendix.

Weaknesses

**Significance** While the overall contribution of the work is impressive, it remains the case that, a) the method is an extension of GraB, and b) not obvious to me that the proposed problem setting is very practical for the considered use cases. It seems to me that much of the difficulty of the analyzed problem is self-imposed: The difficulty in extending the algorithm to the distributed setting is that * the considered problem setting only allows workers to produce permutations of their local data samples… but often in large-scale DDP model training, data can be read by any worker, and is not stored directly on the computing node. This would enable one to simply extend GraB to such settings. * while large step sizes may indeed results in large disparities across gradients at different workers… why not AllReduce the gradients, run a gossip step, or simply average the gradients in the parameter server as is typically employed in DDP? In fact, the proposed algorithm already has an aggregation step at the parameter server. I am also concerned about the overhead of running the PairBalance subroutine in large-scale settings with many workers, especially as the size of model parameters balloons. On the empirical side, I appreciate the breadth of the exploration, but the improvements training LeNet on CIFAR10 seem quite odd. Why should performance of D-RR change with the number of workers, when training with the same batch size?

Questions

Please see questions described in-line in weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations of the work are not adequately addressed.

Reviewer ZrZn7/10 · confidence 4/52023-07-07

Summary

The authors propose a distributed version of Grab. The authors both theoretically and experimentally show that there exists a pattern which can improve training convergence of such a method in a distributed setup. The paper is very well and tightly written with clear limitations. The authors verified their claims both experimentally and theoretically.

Strengths

* The authors have clearly motivated the problem and discussed it in detail how their work compare to other prior works * The experiments are well executed and clearly demonstrate the benefit of CD-GRAB. * The authors discussed the proof in detail in the appendix and organized it well. * All limitations are discussed.

Weaknesses

* As the authors have stated that there is additional memory overhead keeping older gradients. As memory becomes more scarce due to larger models,do authors think there is a way of approximating the previous epoch gradients ? * The loss for baselines seems a bit high, can authors comment on that, for ex- the PyTorch LSTM example on wikitext-2 gives a loss of around 100, as shown here - https://arxiv.org/abs/1905.13727 why the perplexity score is so far off ?

Questions

Compared to D-RR graphs CD-Graph seems to very smooth. Do the authors have an explanation for this ?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have discussed it well.

Reviewer AQrA5/10 · confidence 4/52023-07-10

Summary

The paper introduces Coordinated Distributed GraB, a new method that leverages stale gradients from prior epochs to order examples and achieve a provably faster convergence rate than random reshuffling. CD-GraB demonstrates a linear improvement in convergence rate compared to centralized GraB and showcases superior performance over alternative methods across various benchmark tasks. The contributions of this paper encompass a thorough theoretical analysis of CD-GraB's convergence rate as well as empirical evaluations conducted on three benchmark tasks.

Strengths

1.The paper presents an innovative method called Coordinated Distributed GraB, which builds upon previous work on GraB and kernel thinning. This novel approach capitalizes on the utilization of outdated gradients from prior epochs to prioritize examples, resulting in a demonstrably faster convergence rate compared to random reshuffling. 2.This paper offers a comprehensive theoretical analysis of CD-GraB's convergence rate. The analysis reveals a linear acceleration in the convergence rate of CD-GraB.

Weaknesses

1. This method is designed specifically for the paradigm involving a parameter server and workers. However, it is important to note that as the number of workers increases, there will be a linear increase in overhead, potentially affecting performance during large-scale training sessions. 2. In comparison to the widely used ring allreduce method for gradient aggregation, the communication of gradients between the parameter server and workers in this method is less efficient. Consequently, the improvement in wall-clock time may not be convincing. Furthermore, the unavailability of the code makes it difficult to reproduce the experiments. 3. The convergence analysis presented in the paper may not be easily applicable to adaptive optimization methods such as AdamW, which is a popular choice for training deep neural networks. 4. It is worth noting that the proposed method is based on stochastic gradients, while it is common practice to train models using minibatch stochastic gradients.

Questions

The theoretical analysis in this paper assumes an ideal scenario where the number of data examples is evenly divisible by the number of workers. However, it is crucial to consider the implications when the numbers cannot be perfectly divided or when workers have different numbers of update iterations. How does CD-GraB address these challenges? Is there any theoretical analysis available to support its handling of such problems?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No.

Reviewer dDcg5/10 · confidence 3/52023-07-26

Summary

This paper proposes a method to extend the optimal data order finding method GraB to centralized distributed setting. This is achieved by using an alternative balancing method that does not require using stale averages which would cause a problem given higher learning rates in the distributed settings. Furthermore the ordering task is determined by a global coordinator leading to CD-GraB. The effectiveness is demonstrated empirically over random reshuffling as well as running GraB locally on each worker (with and without the alternative balancing method). Finally, an analysis of the convergence rate is given for non-convex smooth functions as well as functions satisfying PL.

Strengths

The proposed alternative balancing method is novel and can be useful both for the single-node and multi-node versions of GraB. On the experiments CD-GraB is faster both in terms of epoch and wall clock time than random reshuffling. Ablation studies clearly show the importance of using the central coordinator. The paper is written clearly and is easy to follow and understand.

Weaknesses

One of the main drawbacks is that the experiment are quite small in scale. The main comparison is using MLPs and logistic regression while the ablation is done using LeNet instead of the larger ResNet models. For the ablation, the number of epochs is also limited and as a result the obtained accuracy is no where near the state of the art. This makes it hard to judge whether the observed improvement will continue to be there as the training goes on further. Since the paper is advocating for using CD-GraB in practice, large-scale evidence is essential. I would also suggest considering the effect of different architectures including Transformers. If for some reason the method is not applicable to some settings this should be clearly discussed. In addition, in many cases other optimizers such as Adam are used in practice. The possibility of using this method with such optimizers (particularly Adam) as well as its effect on the performance is not investigated which again makes it harder to judge whether it can be directly deployed in practice or not. Moreover, one thing that I would assume can considerably affect the performance in the distributed setting is the data heterogeneity. However, no discussion on this has been made. It would be nice to have at least one set of experiments in the heterogeneous settings. As a side note, since the pair balance algorithm can also be applied in the single-node settings, a good comparison to show the effect of this switch is comparing it with the original GraB in the single node setting, both in terms of speed and the best achievable loss. Though this is covered to some extent given that the performance of local-GraB and CD-GraB match with 4 workers (according to Figure 4), I think a direct comparison in the single node setting would still be useful.

Questions

1. Looking at Figure 3, it seems using GraB allows the optimization to reach a better train loss. Is this true or if the normal RR is run for a longer period it finally reaches GraB's loss? 2. How does using CD-GraB affect the learning rate schedule?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No specific discussion of limitations is included. I think a quantitative discussion around additional resource requirements (e.g. memory) is needed. Furthermore, as a replacement for the current optimization methods, a discussion about its ability to combine with existing solutions in distributed settings is missing. For example, I am not sure if this method can be combined with existing privacy frameworks which could also highlight a direction for future work.

Reviewer ZrZn2023-08-14

Thank you

Reading this response and to other reviewers as well, I believe that this is a good paper. Both from theory and practice point of view. I think the writing is measured and clearly lists the potential shortcomings. I feel the authors are forthright and explicit about their shortcomings. I will vote to accept this paper. I have bumped my score to an accept.

Authorsrebuttal2023-08-16

Response to discussion comment by Reviewer ZrZn

Thank you so much for participating in the discussion period. We very much appreciate it.

Reviewer dDcg2023-08-15

Dear Authors, Thank you for your replies. > We note in the paper/Appendix that we ran we could afford to. CD-GraB converges faster than D-RR, which the existing results show. We are academics, not an industry lab. We can run LeNet for longer, but saved our training budget for other tasks. I completely understand the challenge of limited resources. However, there exists many prior works from academia applying larger models such as ResNet on CIFAR10. I point out that there has been cases where proposed optimization methods fail to reach the high accuracy obtained by random reshuffling GD even though they might be faster in the beginning of training. Therefore, plots showing random reshuffling is outperformed by CD-GraB can become misleading especially for readers not familiar with the expected state of the art results. This is why I find the evidence provided here very insufficient to support practical use of CD-GraB. > Based on the intended scope of our contributions (i.e., primarily theory, with empirical implications to be explored in future systems work), the fairest comparison for our experiments is to the original GraB paper (also a theory-focused paper). We used this prior work as the basis for designing our experiments. In the whole paper, the argument has been always about working both in theory **and** in practice. If this is not the case, this should be clarified and the reason why the optimizer is not practice-ready explained. > We additionally include results for a small GPT-2 on WikiText-103 in the Appendix. We note in the conclusion why we did not focus on transformers. Models like LLMs today are finetuned typically for a really small number of epochs. The benefits of CD-GraB (i.e., getting the theory-backed benefits of achieving the herding bound) only come in the multi-epoch setting, and we cannot afford the pretraining costs of a LLM for a lot of epochs. Thank you. Please add a reference to these experiments in the main text as well. > Even so, we demonstrate the benefits of CD-GraB on other ML tasks, and believe that this is an important contribution. It is worth investing the effort in future ML systems research to build and stress-test a full implementation of an order server at scale. Without the present work, we would not have been able to know that such an investment was worth it. Such a system necessarily is the realm of future work. If I understand correctly, this means that there exist a bottleneck or an obstacle in scaling up CD-GraB which needs a full implementation. This is what I was asking for in the limitation section of my original review. What are these limitations? It does not seem to be the memory based on your explanation. Assuming off-loading to CPU is not needed, what other significant implementation is needed or what part of the algorithm needs to be improved to make it suitable for large scale experiments? > Single node pair-balance Please do add the experiments if you have them. If I am understanding correctly, replacing the ordering method can be beneficial even in the single node setting since it removes the need to store all past gradients. It would be also important to assess whether using this method instead of the original method is less performant in finding the right order or not (which might be hidden in the multi node settings due to incompatibility of the original method with that setting). In general, unless needed, changing only one hyper-parameter (algorithm) instead of both (algorithm and number of workers) allows for better comparison. > Learning rate schedule: We used standard settings for the tasks that we investigated, and for the LSTM on WikiText-2 experiment we still used learning rate scheduler. Investigating how to tune the learning rate schedule in response to our improved convergence is the realm for future theory work. I am not sure what is the standard schedule for LeNet on CIFAR10. A decay in learning rate usually shows itself in a spike in accuracy but I could not find this in the plots. Can you please describe the standard schedule you mentioned? **Regarding the memory**: Thank you for the explanation. I did not understand what is exactly reported in the table. What is meant by overhead? What is the increase in full memory usage (percentage for example)? **Overall comment and suggestion**: My main objection currently is that the evidence presented is not convincing enough for using this method directly in practice which I initially considered to be part of the claim. Still the idea behind the paper is novel and as the authors said the method can open doors for further improvements. My suggestion is that the authors would add a Limitation section clearly discussing all the limitations including the gap between the accuracies on CIFAR10 and state of the art, possible bottlenecks if any (discussed above), etc. to avoid any confusion. Assuming this is possible (please confirm), I have updated my score. Thank you.

Authorsrebuttal2023-08-16

Response to discussion comment by Reviewer dDcg

Thank you so much for participating in the discussion period. We respond below: > I point out that there has been cases where proposed optimization methods fail to reach the high accuracy obtained by random reshuffling GD even though they might be faster in the beginning of training. Therefore, plots showing random reshuffling is outperformed by CD-GraB can become misleading especially for readers not familiar with the expected state of the art results. This is why I find the evidence provided here very insufficient to support practical use of CD-GraB. Thank you for clarifying your initial comment. We observe that we often perform *better* than D-RR, not just at the beginning of training, but overall. If accepted, we will update with revised results/figures that make this clearer. Unfortunately, we cannot do so in the discussion period. > In the whole paper, the argument has been always about working both in theory and in practice. If this is not the case, this should be clarified and the reason why the optimizer is not practice-ready explained. We will update the paper to reflect this. The paper is a theory-based study, which we validate empirically. We scoped it this way to determine the viability of building a fully implemented distributed system. Such a system would be needed to showcase how others more generally could use CD-GraB in practice (i.e., doing distributed training across multiple nodes, if the application warranted this scale, not just within a node with multiple workers). > Please add a reference to these experiments in the main text as well. If accepted, this is what we would use the additional page for. > this means that there exist a bottleneck or an obstacle in scaling up CD-GraB which needs a full implementation. This is what I was asking for in the limitation section of my original review. What are these limitations? It does not seem to be the memory based on your explanation. Assuming off-loading to CPU is not needed, what other significant implementation is needed or what part of the algorithm needs to be improved to make it suitable for large scale experiments? It isn't a bottleneck so much as it requires multiple months of software engineering for a full distributed system implementation. We want to do this work, since we now know from the success of the present work that it is worth the investment to build such a system. We are happy to provide more context on the work needed for such a system in the Appendix. > Please do add the experiments if you have them. If I am understanding correctly, replacing the ordering method can be beneficial even in the single node setting since it removes the need to store all past gradients. It would be also important to assess whether using this method instead of the original method is less performant in finding the right order or not (which might be hidden in the multi node settings due to incompatibility of the original method with that setting). In general, unless needed, changing only one hyper-parameter (algorithm) instead of both (algorithm and number of workers) allows for better comparison. Yes, it is beneficial. We will include these results (we have them from our initial testing of PairBalance, just deleted them due to space). > I am not sure what is the standard schedule for LeNet on CIFAR10. A decay in learning rate usually shows itself in a spike in accuracy but I could not find this in the plots. Can you please describe the standard schedule you mentioned? We used the settings from the original GraB paper (which were in turn chosen from prior work). We will make these choices clearer int he Appendix. > Thank you for the explanation. I did not understand what is exactly reported in the table. What is meant by overhead? What is the increase in full memory usage (percentage for example)? We apologize for this. The rebuttal window closed while we were still making unification edits to our responses. Please refer to the top-level response to all reviewers for a figure (attached as a PDF) that more clearly captures this table. We noticed on the late side that we could include such a PDF with a figure. > My main objection currently is that the evidence presented is not convincing enough for using this method directly in practice which I initially considered to be part of the claim. Still the idea behind the paper is novel and as the authors said the method can open doors for further improvements. My suggestion is that the authors would add a Limitation section clearly discussing all the limitations including the gap between the accuracies on CIFAR10 and state of the art, possible bottlenecks if any (discussed above), etc. to avoid any confusion. Assuming this is possible (please confirm), I have updated my score. We hope we have appropriately confirmed in our answers above. We will also more clearly call out the limitations that we mention in the conclusion using a bolded paragraph heading. Thank you again

Reviewer xnQq2023-08-21

Re: Rebuttal

I acknowledge having read the author response. I have no concerns with this work from a theoretical standing. The authors have also addressed my concerns regrading details of the empirical evaluation. My remaining concern is whether the proposed method work has practical applicability in practice for this community, or whether this is closer to an interesting thought experiment. The difficulty of the problem setting still appears self-imposed to me; I acknowledge the authors' response that the distributed setting is important, but in cluster-based workloads, distributed workers read data from a shared file system, and thus are not artificially restricted to local samples. In terms of gradient disparities, which are a fundamental issue in the considered problem setting, there exist simple strategies in the distributed setting for reducing those (e.g., gossip or communication primitives). I will maintain my weak accept recommendation for now and engage in discussion with other reviewers to asses the impact of extending the GraB method to the setting where a) workers can only shuffle local data, and b) gradient disparities across workers can be large.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC