From promise to practice: realizing high-performance decentralized training

Response to Reviewer LAtq (Part 1/2)

We appreciate the time and effort that you invested in reviewing our paper and giving constructive feedback that will allow us to improve our paper further. Note that our reply is separated in two, to meet the character limit in OpenReview. Please also note that we have updated the supplementary material to include training curves for the GPT-2 experiments and provided more details on the implementation of the extension. We refer to this updated supplementary in our response below. **W1 (related work)**: Thank you for providing us with these references. These are really interesting papers that also focus on practical decentralized training. We would like to emphasize that the application scenario that they worked on and the problems that they tackled are different from ours. In [1,2], the workers are distributed in different cities, and the bandwidth of the interconnections is significantly limited. The compression and sparsification techniques demonstrate more impact in the low-bandwidth scenarios. In our case, as stated in the introduction and in the experimental setups, we focus on self-built servers or shared clusters which may not have high-end interconnections as those dedicated clusters built by big tech companies. Still, their bandwidths are at least 25Gbps, which is much higher than those described in [1,2]. As for problems considered, our paper includes decentralized variants of Adam and its analysis to achieve good performance also on Transformer-based models. Moreover, [1,2] investigated communication scheduling in asymmetric communication environments, which is related to our findings in Section 2.1. In Section 1.1, we discuss decentralized training, several works focusing on practical speedup, and some orthogonal techniques including compression. We believe that [1,2] will be good additions to both "Decentralized training" and "Orthogonal scalability techniques". **W2 (overlapping of comm. and comp.)**: We agree that the idea of using adaptive momentum methods is not novel, and the technical innovation in the convergence proof is indeed limited. However, we do believe that the aggregation mechanism is novel, and it is an essential component in attaining our level of performance in both speedup and generalization. In both image classification and machine translation, the proposed variant of DAdam outperforms DAdam by a margin in generalization performance while maintaining similar speedup performance. As for the overlapping between communication and computation, we are aware that the overlapping by the gradient bucketing technique is widely used, and we also mentioned it at the beginning of Section 2.1. We would like to emphasize that one of our contributions is to further extend this technique the decentralized training. In decentralized training, the model parameters, instead of gradients, are transmitted among workers, and the local updates depend on the information of its neighbors from the last iteration. The removal of the dependency of local updates on the information from the current iteration further enhances this technique, since it allows for overlapping between the forward pass and the communication. As for [3,4], the authors focus on achieving higher computation resource utilization in the context of pipeline parallelism and tensor parallelism, while we focus on data parallelism. The distributed strategies distinguish our paper from [3, 4], but it is certainly possible to combine all of them. In the pre-training of LLMs, for example, it is common to see multi-dimensional parallelism in use (like [DeepSpeed](https://github.com/microsoft/DeepSpeed) and [Colossal AI](https://colossalai.org/)). It will be interesting to see how the optimization techniques in different dimensions can be combined and contribute to speedup at an even larger scale. However, we defer such studies to future work. [1] GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication. In TPDS 2022. [2] Communication-efficient decentralized learning with sparsification and adaptive peer selection. In ICDCS 2020. [3] Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models. In ASPLOS 2022. [4] Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning. In ASPLOS 2024.

Response to Reviewer LAtq (Part 2/2)

**W3 (ablation study)**: Thanks for the valuable comment. We agree that the ablation study will make the presentation clearer to the readers. From the perspective of the necessity of the second-order momentum in the optimizer, we conducted additional experiments with decentralized SGD that we have included in the supplementary material (see Section "Transformer with SGD Optimizer"). We would like to emphasize that we already have the comparisons with the data parallal implementation in PyTorch, stochastic gradiant push, various communication topology, and decentralized Adam with accumulation trick. We believe that we can create a section that performs a cleaner ablation study by combining and re-organizing these results. **W4 (topology)**: Please refer to Table 1 for the communication time of various topologies. The choices of topologies are subject to the structure of the cluster. We are aware of other topologies like the torus graph and the expander graph, but we believe the chosen topologies are suitable to demonstrate advantages in either runtime speedup or convergence rates. In the numerical experiments, we chose `complete` and `alternating-exp-ring` because other topologies may perform worse in both convergence rate and practical runtime. In Section A.8.1 in the appendix, we have results with the one-peer ring topology for the generalization performance on the machine translation task. **W5 & Q1 (convergence behavior on GPT-2)**: Since the training details of GPT-2 are not public, we adapted the code from an open-source implementation from [nanoGPT](https://github.com/karpathy/nanoGPT). In the results of nanoGPT, the GPT-2 (small) was trained for 400k steps, which we further extended to 600k steps and verified comparable generalization performance of decentralized training in our numerical experiments. In the newly uploaded supplementary material, we provide the training curves of training and validation losses. **W6 & Q2 (implementation detail)**: In the section "Implementation Details of Decentralized Training Extension", we have provided examples of how to launch the training and how to use the extension. We have also uploaded the source code of the PyTorch extension in a separate `code` folder. Please refer to the supplementary material for details.

Response to Reviewer pREt

We thank the reviewer for acknowledging the strong performance of our work and the quality of the presentation. Below, we answer the questions and weaknesses that you have raised in your review. Please note that we have updated the supplementary material to include a pseudo code listing that clarifies the idea of the runtime model. We refer to this new supplementary in our response below. **W1 (explanation of the runtime model)**: Thank you for pointing out parts of our paper that you find unclear. We agree that the dependency relationships are hard to visualize. To address your concern, we have annotated the pseudocode for the main worker thread with references to the sequential timeline in Appendix A.5. Please refer to the algorithm in the last part of the section "Implementation Details of Decentralized Training Extension" in the new supplementary material for the pseudocode listing. We hope that this, together with Section A.5 in the appendix, will explain the mechanism in a clearer way. **W2 (connection between runtime model and DAdam optimizer)**: In our paper, we first explored key factors that enable speedups of decentralized training and proposed the runtime model to estimate the influences of these factors on the runtime. One important insight is that the algorithm should follow the combine-then-adapt (CTA) scheme of decentralized gradient descent (see Section 2.1), which is why the decentralized Adam variant in our paper is designed to belong to this family of algorithms. Moreover, speedup and generalization performance are two essential aspects of the practical decentralized training pipeline, and the runtime model and the decentralized Adam optimizer represent two complimentary contributions toward this goal. **Q1 (inspiration of the runtime model)**: To assess the efficiency of our decentralized training extension and ensure that other factors like data loading do not impede the speedup brought by decentralized training, we used the [PyTorch profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) to record computation and communication times in our initial experiments. Through experiments conducted on different hardware setups, we found a strong interplay between the organization of the training algorithm, the communication topology, and the hardware configuration. These observations inspired us to quantify the influence and to design a runtime model to gain insight into tradeoffs and guide the overall design process.

Response to Reviewer oHbd (Part 1/2)

We appreciate the time and effort that you have invested in providing us with constructive feedback. It has been invaluable in strengthening our paper. In what follows, we address all the questions that you raised in your initial review. Note that our reply is separated in two, to meet the character limit in OpenReview. Please, also note that we have uploaded a supplementary material in the rebuttal revision that describes the new experiments with FFCV and provides additional implementation details. We refer to it in our response below. **Q1 (related work)**: Thank you for bringing these papers to our attention. While our work is related to Papers 1–6, it differs in several critical aspects. Notably, apart from Paper 4, which employs first-order momentum, the other papers do not utilize momentum. In contrast, our proposed optimizer is an Adam-based adaptive method that incorporates both first- and second-order momentum, necessitating a distinct algorithmic design and analysis. Moreover, our paper focuses on the design and implementation space, particularly the runtime model, which distinguishes it from these other works. We have already included comparisons with related research in Sections 1.1 and Appendix B.2 of the submitted version, where some of the papers are similar to those you suggested. We will integrate the additional papers that you recommended into our revision. Below, we outline how they can be incorporated. Our algorithm builds upon decentralized gradient descent (DGD) and Adam. Consequently, we have compared it against DGD and state-of-the-art decentralized Adam-based algorithms. Please refer to Section 1.1 and Appendix B.2 for further details. The algorithm by Chen et al. (2023), for instance, adopts a tracking technique similar to gradient tracking (GT) (see line 11 of Algorithm 2). As noted in our introduction of Chen et al. (2023), this tracking technique requires an additional round of communication, whereas our algorithm requires only a single round of communication in each iteration. Papers 1-4 in your review fall into the categories of DGD, GT, and extensions of GT to directed graphs, all of which can be integrated into the comparison in Section 1.1 and Appendix B.2. Specifically, here is an overview of the works you referenced as papers 1-4: Paper 1 proposes a unified theory of decentralized SGD with time-varying topology and local updates. Paper 2 improves the convergence rate of gradient tracking and the relationship with network topology parameters by leveraging new analytical techniques. Paper 3 extends the push-pull algorithm to asynchronous scenarios and establishes a linear convergence rate under the assumption of strongly convex problems. Paper 4 presents a decentralized first-order momentum SGD which employs quantization to reduce communication costs. In Section 1.1 of our submission, we discuss decentralized training and reference practical studies (e.g., Lian et al., 2017; Assran et al., 2019) reporting speedups from decentralized approaches. We also briefly review orthogonal scalability techniques like compression. Papers 5 and 6, which you mentioned, can complement this discussion. Specifically, Paper 6 introduces wait-avoiding group averaging SGD, which uses group AllReduce to limit communication operations within non-overlapping process groups while requiring periodic global synchronization for convergence. Paper 5 extends this with SwarmSGD, a variant of SGD that integrates non-blocking communication, quantization, and local steps, demonstrating convergence through randomized, pairwise communication among nodes. We have highlighted the contributions and focus of this paper in the general response. Combined with the comparisons to related work provided here, we hope to have clarified the position of the paper. [1] Koloskova, Anastasia, et al. "A unified theory of decentralized sgd with changing topology and local updates." International Conference on Machine Learning. PMLR, 2020. [2] Koloskova, Anastasiia, Tao Lin, and Sebastian U. Stich. "An improved analysis of gradient tracking for decentralized machine learning." Advances in Neural Information Processing Systems 34 (2021): 11422-11435. [3] Zhang, Jiaqi, and Keyou You. "Fully asynchronous distributed optimization with linear convergence in directed networks." arXiv preprint arXiv:1901.08215 (2019). [4] Singh, Navjot, et al. "SQuARM-SGD: Communication-efficient momentum SGD for decentralized optimization." IEEE Journal on Selected Areas in Information Theory 2.3 (2021): 954-969. [5] Nadiradze, Giorgi, et al. "Asynchronous decentralized SGD with quantized and local updates." Advances in Neural Information Processing Systems 34 (2021): 6829-6842. [6] Li, Shigang, et al. "Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging." IEEE Transactions on Parallel and Distributed Systems 32.7 (2020): 1725-1739.

Response to Reviewer oHbd (Part 2/2)

**Q2 (statistical difference in losses)**: This is a great question. Indeed, the difference in loss is **not** statistically significant, which supports our conclusion that decentralized training can achieve comparable generalization performance to AllReduce training with the same iteration budget. **Q3 (FFCV)**: Thank you for sharing this framework which also focuses on accelerating the DNN training systems. We would like to emphasize that our contribution in this paper is orthogonal to the idea of FFCV [7]. The FFCV framework accelerates training by removing bottlenecks in data loading and data augmentation. To be specific, the framework achieves its goal by loading the whole dataset into the memory, avoiding the much slower disk IO, and compiling image augmentation and preprocessing operations into high-performance kernels, which alleviates the bottleneck caused by the CPUs. The paper does not contribute to improving the throughput of GPUs, reducing communication overhead, or handling imbalanced workloads, and the distributed strategy of its implementation in [fast training on ImageNet](https://github.com/libffcv/ffcv-imagenet/tree/main) is based on `DistributedDataParallel` of PyTorch which is exactly the AllReduce training baseline that we used in our numerical experiments. Our paper explores the potential of decentralized training in achieving better utilization of computational resources (see Sections 2 and 3), which makes it a better alternative than AllReduce training. Nevertheless, we have integrated our decentralized training extension with the FFCV framework and performed runtime performance comparisons that we report in the recently revised supplementary material. The numerical experiments demonstrate that our decentralized training framework brings better scalability to the FFCV framework. Please see `supplementary.pdf` in the zip file for details. [7] Leclerc, Guillaume, et al. "FFCV: Accelerating training by removing data bottlenecks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.