Communication Efficient Distributed Training with Distributed Lion

Summary

The paper introduces Distributed Lion, a variant of the Lion optimizer, tailored for distributed training environments. Lion, known for its memory and computational efficiency, is adapted to reduce communication costs between workers and a central server. This is achieved by communicating binary or low-precision vectors rather than high-precision floating-point vectors. The paper presents theoretical convergence properties and empirical results that demonstrate Distributed Lion’s robustness and efficiency across various tasks, worker counts, and batch sizes. It shows comparable performance to standard Lion or AdamW optimizers but with significantly reduced communication bandwidth.

Strengths

+ Innovation in Communication Efficiency: The use of binary or low-precision vectors for communication significantly reduces bandwidth requirements, which is a critical factor in distributed training. + Theoretical Validation: The paper provides a solid theoretical foundation confirming the convergence properties of Distributed Lion. + Empirical Evidence: Extensive experiments demonstrate the robustness and efficiency of Distributed Lion across a variety of tasks, making a strong case for its practical applicability.

Weaknesses

- Incompatible with Allreduce: after converting the gradients to binary or low-precision, Allreduce cannot be used for gradient synchronization. One of my concerns is about its communication efficiency in real-world distributed systems, especially training with a high number of workers. - Computation Overhead: While the communication cost is reduced, the overhead of converting updates to binary or low-precision vectors and back might offset some of the gains in certain scenarios. It helps if the end-to-end training throughput comparison is reported.

Questions

What is the fundamental difference between distributed Lion and SIGNUM-like algorithms?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

Large-scale AI model training has increasingly higher requirements on time, cost and environmental impact, so it is crucial to develop efficient optimizers. As an emerging optimizer, Lion optimizer has advantages in memory, computation and sample efficiency compared with AdamW. Distributed Lion: The paper proposes Distributed Lion, which is an innovative adaptation of Lion optimizer in distributed training environment. Using symbolic operations in Lion, Distributed Lion only requires binary or low-precision vectors to be communicated between working nodes and central servers, significantly reducing communication costs.

Strengths

1. Distributed Lion significantly reduces communication overhead by communicating only binary or low-precision vectors between workers, which is particularly beneficial for large-scale distributed training. 2. The paper provides theoretical analysis to prove the convergence of Distributed Lion. 3. Experimental results show that Distributed Lion can achieve comparable performance to the standard Lion or AdamW optimizer while reducing communication bandwidth.

Weaknesses

1. The actual updating on local worker parameters is gradients, while the communicated message is signs. While the theoretical analysis shows this updating can guarantee the convergence, the actual updating style looks like the quantization. The important baselines like QSGD and SignSGD are missed. 2. The performance of Distributed Lion can be sensitive to hyperparameter choices, especially those related to communication and aggregation strategies. 3. The code is not provided. Thus the reproducibility of the experiments is weakened. 4. The experiment performance on the CIFAR-10 is very low. Considering that the well-known validation performance of CIFAR-10 can be achieved as 94%, the proposed results are around 90%. Why the performance decreases? 5. The important baseline SGD with momentum is not provided. 6. The convergence curves on training with ImageNet and OpenWebText are not provided. This makes it hard to identify the convergence speedup between different optimizers. 7. The wall-clock time is not provided. The quantization operation and the majority vote require extra time, it will be better to show this optimizer can reduce the real-world throughputs.

Questions

1. How do you ensure that the hyper-parameters of different optimizers are set as a suitable combination for them? Suitable hyper-parameter settings can ensure the fair comparison. If the above weaknesses and questions are addressed, I'm happy to raise the score.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Two minor points considering the realistic device constraints: 1. The experiment scalability is with 32 GPUs, which is not a large scale distributed training setting. 2. The training model is small-scale with less than 1B parameters.

Summary

This paper extends the Lion optimizer to data parallel distributed training. Unlike optimizers like SGD and Adam, the binary update in Lion can be exploited to minimize the communication. They investigate two cost effective methods for the communication of binary updates; averaging and majority vote. Experimental results show that both methods yield competitive results to global Lion and AdamW.

Strengths

The convergence analysis provided in Section 3 gives some reassurance to this non-conventional optimization method. Results are promising and experimental conditions seem adequate.

Weaknesses

The proposed method is a trivial extension of Lion to data parallel distributed training, so the only interesting contribution seems to be the convergence analysis. The main contribution of this work is supposed to be the reduction of communication overhead, but there are no results showing the actual breakdown of the training time. Therefore, it is not possible to determine whether the reduction of communication volume is actually contributing to the reduction of the overall training time. Since the results seem to vary quite a bit for different models and datasets, such information is useful for determining whether the experiments are conducted for configurations that actually show a significant impact on the training time. There remains a possibility that the current method does not work as well for extremely large models trained with ZeRO 3 data parallelism, which is where the communication overhead really becomes a problem.

Questions

How different are the global binary updates between averaging and majority vote? Are the results similar because they are similar or despite their large difference?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations pointed out above are not explicitly stated in the paper.

Summary

This paper proposes Distributed Lion, a new variant of Lion optimizer for distributed training. The proposed algorithm only requires to communicate binary or lower-precision vectors between workers to the center server, significantly reducing the communication cost. The theoretical analysis proves the convergence of the proposed algorithms. The empirical results show that the proposed algorithms have comparable model performance on CV/NLP applications but with significantly less communication overhead compared to the baselines.

Strengths

1. This paper proposes Distributed Lion, a new variant of Lion optimizer for distributed training. 2. The proposed algorithm only requires to communicate binary or lower-precision vectors between workers to the center server, significantly reducing the communication cost. 3. The theoretical analysis proves the convergence of the proposed algorithms. The empirical results show that the proposed algorithms have comparable model performance on CV/NLP applications but with significantly less communication overhead compared to the baselines.

Weaknesses

1. According to Assumption 3.1, the convergence requires i.i.d. local datasets, while real-world distributed training typically uses non-i.i.d. local data. 2. In the empirical results, there seems to be no wall-clock time for training is reported. Note that the overall goal of communication reduction is to reduce the training time. Thus, it is important to report loss/acc vs. wall-clock time in the experiments.

Questions

1. Is it possible to provide a convergence analysis based on non-i.i.d. data? 2. For the experiments, are the local dataset on each worker i.i.d. or non-i.i.d.? 3. Since the proposed algorithm can compress the communication to an extreme extend, I wonder whether it could also be applied to the federated learning scenario, where the local datasets are not only non-i.i.d., but also highly heterogeneous. 4. Is there any empirical results reporting wall-clock time of training?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are well discussed and addressed in this paper.

Thanks for the responses

I appreciate for the efforts from authors. I have two follow-up questions: - 1. Why does SIGNUM outperform SGD Mom? As a compression method, SGD Mom might be the upper bound of SIGNUM? Maybe the hyper-parameters of the SGD Mom are not well tuned? - 2. Could you give some explanations about why the Optimizer State Communication (ms) of MaVo is half less than the baseline. Because the communication messages are signs, I suppose they should be less than original messages for 16x or 32x.