$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and synchronization locks at scale. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In order to mitigate the increase in communication cost that naturally comes with scaling the number of workers, we introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named $\textbf{A}^2\textbf{CiD}^2$. Our method allows each worker to continuously process mini-batches without stopping, and run a peer-to-peer averaging routine in parallel, reducing idle time. In addition to inducing a significant communication acceleration at no cost other than adding a local momentum variable, minimal adaptation is required to incorporate $\textbf{A}^2\textbf{CiD}^2$ to standard asynchronous approaches. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines and we empirically show that using our $\textbf{A}^2\textbf{CiD}^2$ momentum significantly decrease communication costs in poorly connected networks. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.
Paper
Similar papers
Peer review
Summary
This work discusses the challenges and potential solutions related to training complex Deep Neural Networks (DNNs), particularly regarding the computational and communication demands. The traditional synchronous, centralized approaches to DNN training, while widely used, face limitations in terms of efficiency and scalability, which can be tackled by distributed training methods. Asynchronous and decentralized methods are suggested as they allow for more efficient parallelization of computations and communications, using time-delay fluctuations between workers. Such methods eliminate the need for a central worker to aggregate results, allow nodes to contribute in proportion to their available resources, and use peer-to-peer communication to streamline training. However, the complexity of these methods and the large number of parameters in DNNs still pose considerable communication challenges. The authors address these issues by introducing a novel acceleration method, A2CiD2 (Accelerating Asynchronous Communication in Decentralized Deep Learning), specifically for peer-to-peer DNN training. This method uses pair-wise gossip acceleration, which is largely unexplored for deep learning, and is supported by the analytical framework of Stochastic Differential Equations (SDEs). It decouples computations and communications, requiring minimal overhead and enhancing communication rates. Key contributions of this work include extending the asynchronous decentralized deep learning training framework to non-convex settings, proposing the A2CiD2 mechanism for improving communication efficiency, and minimizing the gap between centralized and decentralized settings in environments with up to 64 asynchronous GPUs. The method has been implemented in PyTorch and will be released as open-source upon publication.
Strengths
The strengths of this paper are as follows: 1: Within the realm of deep learning, the widespread use of High-Performance Computing (HPC) technology has made it possible to achieve exceptional performance in a synchronous and centralized environment. However, not everyone has the means to afford such costly setups, making the development and evaluation of distributed asynchronous algorithms crucial. The authors propose a cost-effective method that significantly enhances communication speed. 2: The A2CiD2 algorithm effectively minimizes the discrepancy between centralized and distributed setups. It functions efficiently in environments hosting up to 64 asynchronous GPUs. This characteristic enhances the algorithm's scalability, rendering it suitable for large-scale machine learning tasks. 3: This study broadens the analytical framework for investigating the design and convergence of these algorithms in non-convex settings. It provides new insights into asynchronous distributed deep learning training. 4: The method proposed substantially improves communication efficiency in distributed learning environments. This could assist in addressing common challenges, such as the straggler problem, synchronization between computation and communication, and bandwidth limitations.
Weaknesses
The weaknesses of this paper can be outlined as follows: 1: While the theoretical analysis provided in Section 3.3 is well substantiated, its tightness with respect to the upper bound remains ambiguous. Further explanation is needed on how the asymptotic convergence corresponds with the experimental results for relatively smaller sizes, like those discussed in Section 4. 2: The study mentions that the proposed A2CiD2 algorithm has been tested with up to 64 asynchronous GPUs. While this is quite remarkable, it may not comprehensively represent the broad array of hardware configurations and scales found in real-world applications. A broader evaluation of the method across different scales and architectures would be advantageous. 3: The paper lacks a clear comparison with other existing distributed training methods, be they synchronous or asynchronous, in terms of computational cost, communication overhead, and performance. Such a comparison could help determine whether the proposed method truly surpasses other methods and under which specific conditions. 4: The paper asserts that training Deep Neural Networks (DNNs) with decentralized methods necessitates substantial communications due to the large quantity of optimized parameters. It remains unclear whether the proposed acceleration method sufficiently addresses this issue and how it compares with other techniques that aim to reduce communication overhead.
Questions
In addition to the issues delineated as weaknesses above, I would be grateful if you could respond to the following queries: 1: In Section 3.1, "Model for a Decentralized Environment," you presuppose the symmetry of communication. Could you elaborate on this assumption and explicate its implications for your model? 2: In Section 3.3, "Theoretical Analysis of A2CiD2," you propose a Poisson process. Nevertheless, the validity of this assumption remains nebulous. Could you furnish some justification or explanation for this assumption? 3: In Section 4, "Numerical Experiments," you cite the utilization of 64 A100 GPUs. Could you provide more details about the network configuration they are connected to, such as Infiniband? The performance of the network itself is likely to affect the results of this experiment, and this information would help elucidate the experimental setup.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
N/A : No pertinent content is found in the paper.
Summary
This work, A2CiD2, proposes a novel decentralized asynchronous training method that incurs only minimal overhead but effectively decouples communications and computations, accelerating pair-wise communications via a provable, accelerated, randomized gossip procedure based on continuous momentum and time. A2CiD2 is also compatible with other asynchronous approaches. Across ResNet benchmarks on image classification, A2CiD2 is able to outperform AllReduce-SGD and AD-PSGD on a cluster of 64 asynchronous workers with A100 GPUs, using various communication network topologies.
Strengths
+. Proposed a decentralized asynchronous training algorithm that outperforms the SOTA approaches +. Demonstrated the advantage of the proposed approach both theoretically and empirically +. Provided code for reproducibility
Weaknesses
-. Writing needs improvement. Hard to follow. -. Missing benchmark: 1. only ResNet is evaluated (e.g., ResNet18 and ResNet50), how about more models like RNNs and larger models like GPT2? 2. only image classification tasks is evaluated, how about more tasks like language modeling? -. Missing details in experiments: 1. what is the network type and bandwidth? 2. how many GPUs per machine? -. Missing evaluation: 1. what is the time-to-accuracy or time-to-loss speedup for AC2CiD2 over AllReduce-SGD and ADPSGD? 2. what is performance of AC2CiD2 with random stranger in the cluster?
Questions
See Weakeness
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
N/A
Summary
This paper proposes an asynchronous gossip-based algorithm for decentralized deep learning by using a continuous momentum. Experiments on real datasets are used for evaluation.
Strengths
1. The studied problem about accelerating communication in decentralized deep learning is interesting. 2. The proposed algorithm accelerates the asynchronous baseline both theoretically and empirically.
Weaknesses
1. The writing can be improved. There are many typos and grammatical errors. Furthermore, there are many informal representations, such as “We demonstrate its efficiency theoretically and numerically; empirically on the ring graph…” in abstract. 2. The proposed method needs to store one more copy of the model, compared with baselines. Hence, it may not be suitable for large models. 3. The experimental results are not convincing. The main baseline for comparison is All-Reduce SGD which has very high communication cost. There have existed many sophisticated decentralized methods, which should be adopted for comparison. 4. This work only considers undirected network topology, which means that the communications are symmetric. But in recent years, directed topology has attracted more and more attention.
Questions
Is the formula between Line 139 and Line 140 correct?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
yes
Summary
This work introduces a new method for decentralized optimization that leverages the notion of continuous momentum to speed up its convergence. The method is justified with theoretical analysis and large-scale experiments on the ImageNet dataset.
Strengths
* The work studies an important problem of distributed training over communication-constrained networks * The results of authors have both theoretical justification and empirical validation in a large-scale setting * Overall, the approach is clearly explained and the contributions are easy to understand
Weaknesses
* The primary disadvantage of the proposed method is its memory overhead. Having an additional copy of model parameters on each worker is quite expensive for models where the memory footprint of the parameters dominates that of the activations (in particular, for Transformer models). As a result, it might be quite difficult to apply A$^2$CiD$^2$ to models beside convolutional networks (for example, to train modern language models), which are a highly popular area of application for distributed training * I feel that the statement in L241 needs to be clarified. All-Reduce methods indeed require more *connections* with the growth in the number of workers, but the total amount of bandwidth is asymptotically independent of the network size for methods like Ring All-Reduce * From my understanding, momentum acceleration for decentralized DL has already been studied in the past (e.g., [1], which has been cited in the submission), although the underlying frameworks are clearly different. I think this work needs to more explicitly distinguish their approach from the QG-DSGDm and other results (e.g., [2]), and ideally include those methods as an additional baseline on top of AD-PSGD [1] Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data. Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich, Martin Jaggi. ICML 2021 [2] SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum. Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat. ICLR 2020
Questions
* In L197, what is the connection speed between the nodes? * What was your motivation for choosing the ring graph topology in the experiments? Is this topology frequently used in practice for peer-to-peer networks? * Is it correct that you consider standard All-Reduce to be a centralized method? (e.g., L80) From my understanding, there is no central worker involved when aggregating updates with this family of algorithms
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
I did not see any explicit discussion of limitations of the work: I would be happy if they mentioned the applicability of their method to models with a larger parameter count (for example, Transformers with hundreds of millions or billions of parameters).
Thanks for the rebuttal with a detailed explanation. The authors have addressed my concerns to some extent through the response, so I will raise the score by one level.
Thank you for participating in the discussion! While we recognize the significance of LLMs in this context (which, we acknowledge, is a crucial concern though not yet a standard and widely established baseline in distributed literature, see e.g. [ [1](https://arxiv.org/pdf/1808.07217.pdf), [2]( http://proceedings.mlr.press/v80/lian18a/lian18a.pdf ) ]), we would like to inquire if you perceive any other gaps or areas that might require further elaboration. We’d be happy to offer additional explanations and insights as needed. Thanks.
How is the theoretical communication cost compared to the synchronous algorithm
Given that all workers are with the same computation time, the asynchronous case reduces to the synchronous case. Then the question is how the theoretical communication cost is compared to the synchronous case [1] which gives the optimal communication bound for decentralized algorithms. [1] Optimal Complexity in Decentralized Training, ICML 2021
Thank you so much for your active participation in the discussion! From a theoretical perspective, we've made significant advancements over the works similar to [[1]]( https://arxiv.org/pdf/2006.08085.pdf ), which exclusively focus on deterministic (and synchronous) algorithms. In [[1]]( https://arxiv.org/pdf/2006.08085.pdf ), the Laplacian matrix $\Lambda$ is derived from doubly-stochastic gossip matrices, ensuring $\Vert \Lambda\Vert=1$ and guaranteeing that every edge spikes precisely once per communication round (resulting typically in $\chi_2\leq 1$, refer to Prop 3.9 of [[2]]( http://proceedings.mlr.press/v202/nabli23a/nabli23a.pdf ) for an in-depth discussion). Generally, deterministic algorithms establish bounds that depend on the spectral gap $\rho = \Vert \Lambda \Vert \chi_1$, and can be accelerated up to $\sqrt{\rho}$. Our novel stochastic (and asynchronous) algorithm, $A^2CiD^2$, goes beyond this by providing the potential to achieve a dependency of $\sqrt{\chi_1\chi_2}\leq \sqrt{\rho}$ theoretically (see the star-graph case for which we’re strictly better and refer to Prop 3.9 and Tab. 2 of [[2]]( http://proceedings.mlr.press/v202/nabli23a/nabli23a.pdf )). To our knowledge, we are the pioneers in achieving such communication bounds for an algorithm applicable to Deep Neural Networks. From a practical standpoint, synchronous algorithms must *wait for the slowest worker at each step*. Asynchronous algorithms come to the rescue by allowing *each worker to operate at its own pace*. Even though they maintain a similar *total* amount of computations (the training stops when a *total number of grad steps* have been done), slower workers contribute less, while faster ones contribute more. When compared to the synchronous case, our basic implementation necessitates less communication and less time (table for ImageNet training with 64 workers): | method | Time (min) | # grad (slowest worker) | # grad (fastest worker) | |-----------------------|------------|-------------------|-------------------| | AR-SGD (Pytorch DDP) | 1.7 $10^2$ | 14k | 14k | | AD-PSGD | 1.5 $10^2$ | 13k | 14k | | AD-PSGD w/ $A^2CiD^2$ | 1.5 $10^2$ | 13k | 14k | Consequently, we demonstrate that the **asynchronous scenarios outperform the synchronous one**: $A^2CiD^2$ represents a pioneering, accelerated, and asynchronous algorithm, grounded in both theory and practicality. We firmly believe that this contributes significantly to the advancement of the field. [1] *Optimal Complexity in Decentralized Training*, ICML 2021. [2] *DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization*, ICML 2023.
Thanks for the response. I am still curious the complete comparison of theoretical communication complexity to synchronous algorithms.
Dear AC, For the sake of simplicity, we will derive the analysis of the communication complexity for the strongly-convex case with full gradient (not SGD), the smooth (and/or stochastic) one being completely analogous. In our paper, we introduce $\Lambda$, the Laplacian matrix weighted with $\lambda_{ij}$, the expected number of times a communication along edge $(i,j) \in \mathcal E$ happens each time unit. Recall that we normalize times so that each node takes one gradient step per time unit in expectation. Thus, *one time unit for us is analogous to one round of computation (one "step") for synchronous methods*. Hence, per time unit, our communication complexity (the expected number of communications) is simply given by $\frac{\text{Tr}(\Lambda)}2$. Synchronous methods such as DeTAG [1], MSDA [2] and OPAPC [3] perform multiple rounds of communications (the Accelerated Gossip procedure in [1, 2, 3]) between rounds of gradient computations by using an inner loop inside their main loop (the one counting "steps"), so that the graph connectivity do not impact the total number of "steps'' necessary to reach $\epsilon$-precision. As **Proposition 3.6** in our paper states $\Vert \bar{x_T} - x^* \Vert^2 = \tilde {\mathcal O} \left( \Vert \bar{x_0} - x^*\Vert^2 e^{- \frac {\mu T}{16 L (1 + \sqrt{\chi_1 \chi_2})}} \right)$, this is analogous to saying $\sqrt{\chi_1[\Lambda] \chi_2[\Lambda]} = \mathcal O (1)$ for our method (i.e., the graph connectivity does not impact the time to converge). Now, let us consider a gossip matrix $W$ as in [1,2,3] (i.e., $W$ is symmetric doubly stochastic) and its Laplacian $\mathcal L = I_n - W$. Then, using $\Lambda = \sqrt{\chi_1[\mathcal L ] \chi_2[\mathcal L]} \mathcal L$ is sufficient for having $\sqrt{\chi_1[\Lambda] \chi_2[\Lambda]} = \mathcal O (1)$. * **synchronous methods**: between two rounds of computations ("steps"), the number of communication edges used is $\frac{\vert \mathcal E \vert}{\sqrt{1 - \theta}}$ with $\theta = \max \\{ \vert \lambda_2 \vert , \vert \lambda_n \vert \\}$ the eigenvalues of $W$. * **ours**: the number of communication edges used per time unit for our method is $\frac{\text{Tr}(\Lambda)}2 = \frac 1 2 \sqrt{\chi_1[\mathcal L ] \chi_2[\mathcal L]} \text{Tr}( \mathcal L)$. As, in [1,2,3], each communication edge is used at the same rate, we can apply **Lemma 3.3** of [4] stating: $\sqrt{\chi_1[\mathcal L ] \chi_2[\mathcal L]} \text{Tr}( \mathcal L) \leq \sqrt{\Vert \mathcal L \Vert \chi_1[\mathcal L] (n-1) \vert \mathcal E \vert }$. We have: * $W$ is stochastic: $\Vert \mathcal L \Vert \leq 2$ * the graph is connected: $n-1 \leq \vert \mathcal E \vert$ * definition of $\chi_1$ and $\theta$: $1 - \theta \leq \frac 1 {\chi_1[\mathcal L]}$ Thus, $\sqrt{\chi_1[\mathcal L ] \chi_2[\mathcal L]} \text{Tr}( \mathcal L) \leq \frac { \sqrt{2} \vert \mathcal E \vert}{\sqrt{1 - \theta}}$, which **proves that our communication complexity is better than any accelerated synchronous method**. In some graphs, the difference can be arbitrarily big as shown with the following table: | Method | Comm. complexity per “step”/time unit | Star Graph | Circle Graph | Complete Graph | |-----------------------------------------|---------------------------------------------------------------------------------------------------------------|------------|--------------|----------------| | Accelerated Synchronous (e.g., [1,2,3]) | $\frac{\vert \mathcal E \vert}{\sqrt{1 - \theta}}$ | $n^{3/2}$ | $n^2$ | $n^2$ | | Ours | $ \sqrt{\chi_1\chi_2} \text{Tr}( \mathcal L)$ | $n$ | $n^2$ | $n$ | We will add those elements to our paper, and we really hope that we answered in a satisfactory manner to your concerns about the optimality of stochastic (asynchronous) algorithms. Note that the fact that stochastic optimization algorithms are faster than their deterministic counterparts is something well known in centralized settings (e.g., [5]). [1] *Optimal Complexity in Decentralized Training*, ICML 2021. [2] *Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks*, ICML 2017 [3] *Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization*, NeurIPS 2020. [4] *DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization*, ICML 2023. [5] *Tight Complexity Bounds for Optimizing Composite Objectives*, NIPS 2016.
Thank you for your prompt responses, despite the limited time frame. I grasped the main points from the replies, but they didn't address my questions. It would have been helpful had you clearly outlined the limitations from the outset. I found there to be more limitations than I had initially anticipated. The asymptotic behavior of the algorithm for very large data sizes and its practical behavior for small, limited data sizes should be discussed systematically. The dominant terms might vary depending on whether the data size is large or small. While it's reasonable to assume that the algorithm's theoretical properties contribute to its positive experimental outcomes, various factors influence the calculations. Regarding Weakness 2, the objective isn't to conduct numerous experiments across multiple architectures, but to derive a broader spectrum of insights and predictions with a limited set of experiments in various environments. This topic is frequently broached in fields like HPC (High Performance Computing), making such discussions quite beneficial. If there's a model addressing computation, data, and communication quantities, a more comprehensive discussion and set of findings could emerge. I understand the challenge of presenting extensive information within a paper of limited length. However, after revisiting the experimental section, it seems, from a reader's perspective, to be primarily a performance evaluation within a constrained setting. Nonetheless, I'm not disputing the value of your method.
Thank you very much for participating in the discussion. * There seems to be some misunderstanding about the theoretical analysis. *We don't adopt an asymptotic approach*, as we are not examining the scenario where "t" tends towards positive infinity. Instead, we focus on a quantitative approach, considering specific time steps “t”. Additionally, it's important to note that *the sizes of the data/datasets do not influence or factor into our convergence analysis*. We believe our algorithm's theoretical attributes are well mirrored by our experimental outcomes. If the AC allows us to, we will illustrate this match between our theoretical convergence rate and the observed one with a Figure (this is not surprising given that communications here are "convex," capitalizing on the corresponding theory). We kindly ask the reviewer to clarify what additional factors precisely they believe could be better addressed or clarified in the theoretical analysis. * We’re slightly unclear on the expectations of the reviewer, as our work is built upon the standard norms in the community, both from the theoretical point of view (see [1,2,3,4,5,7], even for the Poisson Process use), and from the experimental set-up perspective ([5,6,7,8] leading to our implementation using 64 GPUs, ResNet model, CIFAR10/ImageNet datasets with the objective to maintain performance under the decentralized constraints). It would be very helpful if the reviewer could provide specific references. We thank you in advance for any further clarifications. [1] *Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks*, ICML 2017 [2] *Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization*, NeurIPS 2020. [3] *DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization*, ICML 2023. [4] *A Unified Theory of Decentralized SGD with Changing Topology and Local Updates*, ICML 2020 [5] *Stochastic Gradient Push for Distributed Deep Learning*, ICML 2019 [6] *Don’t use large mini-batches, use local SGD*, ICLR 2020 [7] *Asynchronous Decentralized Parallel Stochastic Gradient Descent*, ICML 2018 [8] *Consensus control for decentralized deep learning*, ICML 2021
Thank you for your reply. As a result of the discussion so far, I would like to raise my rating by one step, but I still think the biggest problem with this paper is the lack of clarity in the structure of the paper. As I have already pointed out, I think the connection between the theoretical and experimental analyses is poor. There are many gaps between the theoretical and experimental results, and I think it would be better to increase the experimental part even if the amount of the theoretical analysis part is reduced. I think it would have been better to conduct separate measurements for the calculation part and the communication part, not only to show the overall superiority of the method but also to go into the characteristics and reasons for the superiority of the method. I don't think there are any particular problems with the choice of the computing environment or data set. Again, the lack of experimentation and discussion is problematic.
We greatly appreciate your decision to increase the score by 1 and we thank you for acknowledging that there is no problem with our computing and datasets choices. We have produced an additional analysis (which we have shared with the AC) that empirically compares standard gossip with $A^2CiD^2$ solely from a communication standpoint, demonstrating a strong match between theoretical communication rates and empirical consensus convergence speed. We also want to re-emphasize that our experiments include training DNNs, using one of the most challenging graph (a ring graph), conducted on 64 workers, on Imagenet, which is strong evidence of the benefits of our method from an experimental perspective. Given the theory, demonstration of its match to practice, and the evaluations in challenging settings we kindly urge the reviewer to reconsider their evaluation. Thanks for your attention.
Thanks for the response
Thank you for the response. The authors have addressed most of my concerns. Hence, I raise my score by one level. The issue about experiment has not been addressed. AD-PSGD is relatively outdated. More advanced baselines should be adopted for comparison, such as DADAO (ICML 2023) and other methods cited in the reference list. DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization, ICML 2023.
AD-PSGD is the state-of-the-art baseline in asynchronous decentralized DNN training
Unfortunately, DADAO is designed and evaluated for the convex case and isn't suitable for training DNNs as it relies on the saddle points of a Lagrangian, which has no equivalent in Deep Learning. Indeed our preliminary experiments for this paper found DADAO obtained poor performance for DNNs in practice and it was excluded for further consideration. To the best of our knowledge AD-PSGD stands as the state-of-the-art baseline for asynchronous decentralized DNN training algorithms (see also reply to reviewer fkqi), thus we ask the reviewer to reconsider their assessment regarding the strengths of the baselines.
Decision
Accept (poster)