Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)

Summary

this paper proposes a gradient descent technique for learning deep neural nets with large batch sizes. the authors focus on the setting where the model size is small to medium (10M to 300M parameters) and the dataset size is also small to medium (up to 1M images in vision or ~ 3B words in NLP) but with as large batch size as possible. the goal of this works to accelerating the learning of such models(e.g., ResNet on Imagenet, BERT on the original BERT-used corpus) by using less training steps and less training time. The authors were able to show the proposed method can achieve good results (without big drop in eval metrics) while using larger batch sizes than prior arts. And when the proposed method is compared with prior arts at the same batch size, the proposed method seems to outperform prior arts as well.

Strengths

The papers covers standard benchmarks like ResNet on ImageNet and BERT pretraining. and therefore can be fairly compared against many prior arts on the same tasks

Weaknesses

This paper claims "training acceleration" as a key contribution. But throughout the paper, the comparison on speed up is based on number of steps or number of epochs. It is unclear what are the speed advantage in terms of wall-clock time by using the propose technique. I also checked the supp. pdf. In other works (such as LARS and LANs, which are cited by this work), the authors usually report actual wall clock time speedups as they increase the batch size and the compute infra. It was disappointing to not see any mention of that given the authors are using 768 GPUs (therefore I expect very interesting scaling behaviors)

Questions

* how does the proposed method help with the large scale trainings such as CLIP models or DiNOv2 models, which the datasets are much bigger than traditional settings like ImageNet? * what would be an ideal scenario for using the proposed technique in a computer vision task? from Table 2, there is a clear trade-off between accuracy vs batch size. * in Table 1 and Table 2, how much time does each training take (under different batch sizes)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

the proposed technique is only validated on small-medium size model on small-medium size dataset. it is not applicable (at least no evidence provided) to large scale model training (either large in dataset size or large in model size.

Summary

The authors strive to propose a heuristic training strategy, called variance reduced gradient descent technique (VRGD), based on the gradient signal to noise ratio, i.e. the ratio between the norm and the variance of gradient. Compared to vanilla training, VRGD scales the learning rate with the gradient signal to noise ratio during each iteration. Then, the authors prove that the proposed method can converge within finite training steps, and claim that it will give better generalization gap compared to the vanilla training. Finally, the authors show the effectiveness of their methods via Bert-training and ImageNet training with up to 96k batch size.

Strengths

***Strengths*** 1. The paper is clearly written and easy to follow. 2. With the development of hardwares, training with large batch would gradually become a basic requirement for training large-scale models on gigantic data, such as GPT. In my opinion, the authors are focusing on a very topic worthy to probe.I believe that this paper may have a potential significance not only in academia but also in the industry. However, some modification may be required currently. 3. The authors show some interesting and valuable experiment results, but not comprehensive enough.

Weaknesses

***Weakness*** 1. (Major) The authors repeatedly mention that large batch training can lead to sharp minima in Abstract and Introduction, which seems to suggest that the proposed method can avoid such problem. However, I have not found discussions or observations regarding the proposed method can solve this issue. So can the proposed escape these bad minima? Considering that many recent works that focuses on guiding training to converge to flat minima i.e. SAM family/gradient norm regularization, I am quite curious what would be like to adopt the proposed method in these algorithms when given large batch training. And from the results, I find that using the proposed method alone would not result better performance than these flat-minima-based methods. I have listed some typical works below. [1] Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." ICLR2020. [2] Kwon, Jungmin, et al. "Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks." ICML2021. [3] Zhuang, Juntang, et al. "Surrogate gap minimization improves sharpness-aware training." ICLR2021. [4] Zhao, Yang, et al. "Penalizing gradient norm for efficiently improving generalization in deep learning." ICML 2022. 2. (Major) Continuing with the previous comment, I find the proposed method is rather heuristic, lacking a clear motivation. I could not find clear rationale and sufficient analysis that the proposed method can benefit training. Essentially, the proposed method is simply to scale the learning rate adaptively based on a specific parameter (Eq. 10). For me, I am not quite convinced that such a learning rate scaling policy will lead to reasonable performance gain. Meanwhile, the author argue that the proposed method can lead to smaller generalization gap. However, the core that supports this claim is based on an empirical observation, which makes the mathematical proof not quite rigorous and the claim much weaker and unhelpful. 3. (Major) I would like to discuss the convergence of the proposed method. Firstly, to my understanding, the convergence analysis focuses on analyzing to what extent can training converge on the given training samples, not testing set. So, it is not quite appropriate to use the convergence curve on the testing set to demonstrate the conclusion regarding the convergence analysis. i.e. Figure 2. Secondly, from Figure 2, the authors state that the proposed method converge 1.7~4 times faster than the conventional optimizers. But, I could not observe such a big gap between them from Figure 2, so could the authors explain how to measure the convergence here. Thirdly, a tighter bound in convergence would not give any information regarding the testing performance. A looser bound and slower convergence rate can give better testing performance in many cases, for example SAM. The author can refer to the paper below. [5] Andriushchenko, Maksym, and Nicolas Flammarion. "Towards understanding sharpness-aware minimization." ICML 2022. Note that I am not saying a faster convergence is harmful. In my opinion, the core meaning of this convergence section is to prove that the proposed method can converge in finite time. And it is not surprising that The proposed method shares the same convergence rate as that in SGD, i.e. O(1/sqrt{T}) given that this method is to scale the learning rate compared to SGD. In the current version, it appears that the convergence section seems to heavily imply that the proposed method outperforms SGD without a promise of testing performance, which I disagree with. 4. (Minor) Line 32. "However, Keskar et al. [2017] theoretically analyze the LB training and finds that it can be easily trapped into sharp local minimum, leading to strong generalization gap". Actually, the cited paper (Keskar et al. 2017) has not provided theoretical analysis. 5. (Minor) Line 95. To my understanding, the variance of random vectors is a matrix, i.e. covariance matrix. Why is a scalar here? 6. (Minor) It is highly encouraged to show the results of training vision transformers with the proposed methods.

Questions

See weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I have not found any discussions about the limitations and potential negative societal impact. But in my opinion, this may not be a problem, since the work only focuses on the learning method in deep learning. Still, it is highly encouraged to add corresponding discussions.

Summary

This paper examines the improvement of training throughput in a large batch parallel training setting. By employing the gradient signal to noise ratio (GSNR) as a measurement of the generalization gap during training, the authors introduce a variance-reduced gradient descent method designed for large batch training scenarios. The authors provide theoretical analysis to substantiate that the proposed variance-reduced gradient descent (VRSGD) method exhibits superior generalization compared to stochastic gradient descent (SGD) and potentially achieves faster convergence. Additionally, experimental evaluations on BERT pretraining, ResNet training, and DLRM training are presented to demonstrate the superiority of the proposed method over alternative approaches for large batch training. Furthermore, orthogonal experiments and analyses pertaining to the behavior of GSNR and sensitivity to hyperparameters are conducted to further support the superiority of the proposed method.

Strengths

1. The research topic of large batch training is of considerable interest. 2. The proposed method is both simple and effective for large-batch training. 3. The theoretical analyses of the convergence rate and generalization are persuasive. 4. The proposed method consistently demonstrates improvements over other baseline approaches. 5. The paper is well-written and easy to understand.

Weaknesses

1. The contribution is limited. It seems that the authors just introduce GSNR into the large batch training, but do not highlight why GSNR is important to large batch training. 2. Although the theoretical analysis and experimental results presented in the study are persuasive, the rationale behind the necessity of the GSNR method for large-batch training remains unclear. It is essential to provide a more comprehensive explanation of why GSNR is specifically relevant and advantageous in the context of large batch training. While GSNR can potentially enhance generalization in various settings, a more explicit justification is required to elucidate its particular suitability and effectiveness for large-batch training scenarios.

Questions

NA

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

The paper proposes a new method for large batch training. It is based on the insight that the gradient-to-signal-noise-ratio for each parameter should be reflected in its learning rate, and hence modifies gradient descent to reduce the variance of the gradients. The paper then shows convergence rates of VRSGD, which are the same as SGD asymptotically, and states that VR-SGD is particularly suited to large batch training where the GSNR will be high. The paper also shows that the proposed method has a smaller generalization gap than SGD in the large batch setting. Experiments are then performed to support these claims on a variety of tasks and architectures.

Strengths

1. The method is based on a solid insight. 2. The empirical results indicate decent gains over baselines.

Weaknesses

1. The assumptions of smoothness and bounded gradients seem a bit too strong.

Questions

Can the assumptions for the theoretical analysis be relaxed?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See weaknesses.

Summary

This paper focus on using large-batch training to accelerate the training of neural network. Specifically, the authors try to use variance reduced gradient descent technique to scale up the batch size and therefore to accelerate the training. The experimental results illustrate that the proposed method can scale up to larger batch size and further accelerate the training of ResNet, BERT and DLRM.

Strengths

Strength: 1. This paper focuses on an important problem, accelerate neural network training. especially for large-batch training. 2. The proposed method is very easy to follow. 3. The authors provide some results to verify the performance of proposed method.

Weaknesses

Weakness: 1. The authors can provide more visualization and analysis about the proposed method and why the proposed method can further scale the batch size. For example, the proposed method can help the model converge to s flat region? 2. The authors should provide more results about the wall time of each experiment and verify whether the proposed can save the training time. 3. I'm not sure whether you should compare your method with more baselines since LARS and LAMB is not current SOTA.

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Response

I thank the authors for their response, and stand by my review.

Thanks for the kindly response.

Thanks for the kindly response and all the authors' effort. I think the proposed method is interesting, and believe it may have potential impact in learning method. However, I think the current version is not ready to publish. Based on the authors' response, I decide to keep my rate. 1. I can understand the authors' intention. The uploaded pdf again has conveyed the idea that the proposed method would encourage training towards flat minima.However, firstly, I could not find sufficient view regarding why the proposed method would avoid sharp minima, especially considering that unlike SAM, the proposed method does not target to guide the training towards flat minima in a straightforward manner. Importantly, given the method targets to improve generalization from the perspective of flat minima training, the authors have not provided any comparison or show the compatibility with the SAM family learning methods, which is considered as the current dominating method in regards to this problem. Meanwhile, based on the experiments and my own training experience, the proposed method would not give better results compared to SAM family (96.52 with SAM using ResNet18 on Cifar10 [1], the best acc is 93.79 with the authors' method using ResNet56). And the authors argue in their rebuttal that "We checked the results of SAM family/gradient norm but found that the batch size of their experiments is no more than 4k, which is smaller than our proposed method." However, this does not mean that SAM could not apply to large batch training. Several papers have already presents the results of SAM training with large batch. For example, [2] provides results of SAM training with up to 32k batch size, against what the authors are arguing. So, in my opinion, a basic comparison between the proposed method and SAM is minimum for acceptance. [1] Efficient Sharpness-aware Minimization for Improved Training of Neural Networks, ICLR2022 [2] Towards Efficient and Scalable Sharpness-Aware Minimization, CVPR2022. 2. Essentially, the proposed method is simply to scale the learning rate adaptively based on a specific parameter (Eq. 10). The core that supports why such training can give better generalization gap is based on an empirical observation, which is quite not rigorous and useful. As far as I know, the mathematical discussion of generalization bound should only be based on some reasonable basic assumptions, like L-smoothness, etc. Meanwhile, the authors have not give clear response in their rebuttal. 3. Based on the results, I could observe that the proposed method presents faster convergence rate for the given case, but the metric that the authors use to gauge the convergence rate is questionable. The authors use the epochs that different optimizers reach the same acc in the middle of the training (in the authors' rebuttal, optimizers reach 0.48 acc), as the speed of convergence. To my understanding, convergence rate is the speed that reaches optimal solution. However, the authors' metric firstly uses the middle state of training, not the optimal solution, and secondly, has not gauged the "speed", i.e. variations of values.

Official Comment by Reviewer FtDV

Thank you for the authors' response. Following a thorough review of the rebuttal, some of my initial concerns regarding the significance of the GSNR in the context of large-batch training have been alleviated to a certain extent. It is now evident that GSNR is helpful for generalization and convergence when dealing with larger batch sizes. In light of this clarifying information, I have chosen to retain my original evaluation score.

Thanks for your response!

Thanks for your response! 1. I hope to see more clear visualization about loss landscape, such as the figure in this paper [1]. 2. Transformer model is easier to converge to sharp local minima compared with CNN. If you focus on reducing generalization gap, maybe you should provide an analysis about generalization on Transformer model, such as vision transformer or some other nlp models. 3. I'm happy to see the authors provide the training time results in the attached pdf. That will make the paper stronger. 4. I still find the performance drop of your proposed method in table 2 when scaling the batch size to 32k, for example, from 4k-77.23% to 32k-76.81%, but there is no drop from 4k to 32k for LARS. In my past experience about large batch training, 32k is not the bottleneck for imagenet training and it usually doesn't occur drop when scaling batch size to 32k. You can also find the results in original LARS paper. Could you please provide some explanation about this phenomenon. If GSNR can works well for large batch training, the performance drop should occur when the batch size is larger than 32k. [1] Visualizing the Loss Landscape of Neural Nets