No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Summary

This paper comprehensively evaluates three efficient training algorithms for transformer language models: layer stacking, layer dropping, and selective backpropagation. Such an evaluation is crucial due to the extensive computational resources needed for training transformer-based models. The authors highlight the importance of specifying a training budget in performance comparisons. They introduce a measurement called "reference system time" (RST) to facilitate comparisons across different hardware configurations. The study examines these algorithms under various training budgets, model architectures, and datasets. The main finding is that these efficient training methods do not always significantly improve over the standard training baseline. Layer stacking tends to outperform the baseline and other methods in most instances. However, layer dropping sometimes falls short, while selective backpropagation often diminishes performance. The paper also explores how these algorithms perform in a downstream task setting, showing that pre-training performance does not necessarily correlate with the ability to generalize in these tasks. It concludes by stressing the importance of caution when using these efficient training methods due to their potential overheads. Further implications of this study can guide the choice of efficient training methods in transformer-based language models by considering the trade-off between improved performance and computational resources.

Strengths

1. Originality: The paper's focus on evaluating efficient training algorithms for transformer language models under a defined training budget is a novel perspective. The introduction of "reference system time" to compare training times across different hardware configurations shows ingenuity. 2. Quality: The paper exhibits high quality in its methodology, experimental setup, and analysis. The experiments are thorough and well-conducted, involving different parameters like training budgets, model architectures, and datasets. The findings are meticulously analyzed, providing an honest assessment of the capabilities and limitations of the methods. 3. Clarity: The paper is well-written and logically organized. The authors clearly explain the motivations behind their work, the reason for choosing specific efficient training algorithms, and their experimental methodology. They discuss their findings in an understandable and straightforward manner. 4. Significance: The paper notably contributes to training language models. Its focus on efficient training methods and the introduction of RST offer practical solutions to the computational challenges researchers face. This research opens the door for more studies on efficient training of such models. The paper's findings and suggestions are invaluable for researchers and will aid in making informed decisions when implementing efficient training strategies.

Weaknesses

1. Limited Scope: The evaluation in the paper is limited to three specific efficient training algorithms (layer stacking, layer dropping, and selective backpropagation). 2. Limited Novelty: This paper mainly focuses on revisiting methods in a specific context. There are not many novel ideas or groundbreaking discoveries introduced in the paper. Studies expected to be published in top-tier conferences like NeurIPS should ideally contain significant novel contributions. 3. Lack of Rigor in Experimental Design: While the authors propose using Reference System Time (RST), they do not sufficiently justify its superiority over other potential measures of computational effort. 4. Overemphasis on Computational Efficiency: The authors focused on computational efficiency, sidelining the quality or performance of the models. This imbalance in focus can give a skewed perspective.

Questions

1. It would be beneficial to understand how the proposed Reference System Time (RST) handle variations in software optimization levels for different algorithms, which might influence the perceived efficiency of other methods. 2. For the RST measure, how much does the reference system time vary across different systems and configurations? 3. The paper asserts that the algorithms provided 'little effect' or 'marginal improvements'. Can you clarify what you consider to be substantial or significant improvements in efficiency and why the results obtained didn't meet that criteria? 4. In some comparison studies with baseline training, it's unclear whether the hyperparameters for the compared methods and the baselines were tuned independently. It would be beneficial if the authors clarified this aspect. Moreover, further details regarding the methodology behind establishing the "budget-adjusted baseline" would be appreciated.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have clearly demonstrated some limitations in their paper, specifically highlighting that their study focuses on a select few efficient training algorithms and acknowledging that further exploration of additional algorithms is required. Furthermore, the authors admit that their review is primarily centered on language model pre-training and that the findings may not be necessarily applicable to fine-tuning or other data-intensive modalities. However, the authors could have expanded on potential negative societal impacts. For instance, if these inefficient training algorithms are utilized for large-scale projects, it could lead to unnecessary usage of computing resources, subsequently affecting energy consumption and carbon emissions. Additionally, the misrepresentation of algorithm efficiency could potentially mislead the academic and industrial communities in their pursuit of efficient models. To improve the paper, it would be beneficial for the authors to provide more context on possible wider implications, particularly focusing on the ecological impact and potential misdirection of research efforts. They might also consider suggesting some directions for future work in improving the efficiency of training algorithms, both in terms of reducing computational costs and minimizing environmental impact.

Summary

Many algorithms have been proposed to make the training of ever larger models more efficient. The authors present a critical empirical study of three selected training algorithms (layer stacking, layer dropping, and selective backpropagation) with fixed training budgets and find that these algorithms often do not make BERT and T5 pre-training significantly more efficient.

Strengths

This paper re-visits evaluation standards, and offers a careful, rigorous, and clean re-evaluation of three efficient training algorithms. It is an excellent example for the importance of also publishing "negative" results (no gains in metrics, no new algorithm). Specifically, this finding can save development time in the implementation and maintenance of these algorithms and reduce the complexity of the pre-training algorithms.

Weaknesses

I agree on the limitations pointed out by the authors in the section "Limitations and Future Work", including evaluation of small subset of efficient training algorithms only, language model pre-training only. Overall, the paper might be felt to be too simple and straight forward, the more as it addresses a well-known issue in a noisy field.

Questions

I agree with the authors that "As illustrated in Section 5, there is an abundance of efficient training algorithms, and rigorously evaluating all of them is prohibitively expensive." (Section 6). What would be an alternative approach to avoid such explosion of algorithms and experiments?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, see the section "Limitations and Future Work".

Summary

The paper reevaluates several training algorithms aimed at enhancing the efficiency of Transformer-based models, such as layer stacking, layer dropping, and selective backpropagation. The authors effectively manage the training resources by employing a metric called reference system time. However, the key result of the study indicates that these three efficient training algorithms yield only modest improvements compared to standard training methods.

Strengths

- The authors conduct a thorough evaluation of efficient training algorithms, providing valuable insights into their effectiveness and highlighting the marginal gains such methods achieve. This revisit contributes to a deeper understanding of existing methods and offers a practical perspective on their applicability. - The authors point out the shortcomings of using wall-clock time for reference, and instead propose reference system time (RST) for a better estimate, which is valuable rule to highlight.

Weaknesses

- While revisiting these methods undoubtedly holds value for the research community, it is important to note that the obtained results may be somewhat trivial and lack significant insights. The paper might be better suited for more specialized venues. - While the primary focus of the study lies in evaluating the original approaches, providing additional discussions on methods and evaluations that build upon these evaluated techniques would offer a more comprehensive understanding of how these methods have been adopted and evolved since their initial proposal.

Questions

Do the results of this study depend on the size of the models? Can we anticipate that the effectiveness of these efficient training techniques would be more diminished or enhanced when training larger models with a greater compute budget?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

In Section 6, the authors acknowledge the limitations of their work and highlight that it is not feasible to evaluate all types of training efficient algorithms due to the potentially exorbitant costs involved.

Summary

This paper studies three efficient pertaining techniques for Transformer models (layer stacking, layer dropping, and selective backpropagation). At variance with previous works on the subject, the authors adequately control for the learning rate schedule by evaluating performance at fixed compute budgets (as defined through a scaled wall-clock time). In doing so, they find that most of these methods provide unclear gain over a vanilla baseline -- and this finding holds across T5/BERT-like models.

Strengths

* **S1.** At variance with previous works, the authors adequately control for the effect of the learning rate schedule. This is a common mistake in countless papers, which should be more discussed as it has previously impacted prominent work around scaling laws (see Hoffmann et al., 2022 as a response to Kaplan et al., 2020). * **S2.** This paper provides negative results on the timely topic of efficient pretraining of Transformers, allowing the community to take a step back and retrospectively analyse the validity of previous findings. Under that light, negative results can be as valuable as positive ones. * **S3.** (minor) The color scheme (which is consistent across text & plots) is a good idea to visually help readers.

Weaknesses

* **W1. Limited significance due to limited adoption of the practices evaluated.** The impact and significance of this work is limited, as it evaluates three methods which have not been widely adopted by the community. For a negative result paper to be truly valuable, it is better if it addresses a common practice rather than a result that has not gotten traction anyway. * **W2. It is unclear how the proposed Reference System Time approach improves upon other methods.** The authors bring-up RST as one of the significant contribution of the paper, and base their analysis on its usage. However, it's unclear how it differs in practice from simply using wall-clock time -- especially since the authors don't actually perform cross-hardware comparisons, which would be one of the benefit of formalising RST. * **W2.1** The authors claim "Unfortunately, WCT can fluctuate even on the same hardware, for instance, due to the usage of non-deterministic operations, hidden background processes, or inconsequential configurations, such as the clock rate". It is not clear how RST improves upon this -- the first recording used to scale RST may be noisy as well, and so could the recordings being scaled. * **W2.2** It is unclear whether Figure 1 is based on real measurements or is simply there for illustrative purpose. * **W3. The results are disparate, incomplete, and lack clarity.** The results are middling, lacking a clear finding (overall it seems layer dropping is worth it across all budgets for BERT, layer stacking may bring gains for longer training budgets on BERT but only on shorter budgets for T5, and that selective back propagation is never worth it). For a paper with negative results, clarity is key to improve upon past work. * **W3.1.** Selective back propagation is compared in a completely different setup, on a single budget, and downstream performance is never measured for SBP+T5 -- only validation loss. Furthermore, while SBP+BERT are trained on 3 different datasets (a rather interesting study given the nature of SBP), only validation losses are provided for comparisons (no downstream task performance) and no conclusion is proposed on the influence of data source on SBP. The lack of a unified framework for comparisons between layer stacking/dropping and SBP make the paper feel more like a pot pourri of ideas rather than a principled & systematic comparison. * **W3.2.** Reporting training loss instead of validation loss in Figure 2 is questionable: as the authors later discuss in their ablations in Figure 5.b), layer dropping acts similarly to drop out -- so it makes sense it is behind on training loss in Figure 2. Validation loss here would be far more valuable. * **W3.3.** The figures are not clear: the x-axis in Figure 3 is not specified, the legends lack an analysis/mention of the key results for each figure, it's impossible to see differences in Figure 4, etc. Specifically to Figure 4/Table 1, the authors mention the concept of Pareto front multiple times in the main text; maybe plotting it would be far clearer than the figures/table proposed for downstream evaluation. * **W4. Insufficient clarity and lack of added educational value.** Negative results papers can prove themselves especially valuable if they help correct a bad practice in the community; here, a general lack of clarity and of adequate framing make this paper fall short of being an educational read. * **W4.1.** The introduction of RST is confusing and frankly not necessary, as discussed in W2. * **W4.2.** The issue around learning rate schedules could benefit from additional discussion & illustration. In particular, it would be interesting to mention that this is the crux of the difference between the scaling laws of Kaplan et al., 2020 and Hoffman et al., 2022. Not accounting for the influence of the LR schedule caused the Kaplan work to inappropriately recommend scaling model size primarily, instead of scaling model and data jointly. * **W4.3.** The paper contains numerous imprecisions and overall lacks clarity (see W3.3. as well). The authors sometime abuse the citations by citing far too many work at once instead of the most relevant ones (l14, l32, l78, etc.). The default citation style of NeurIPS doesn't help here, making it difficult to identify the works at a glance... l144 it's unclear for instance what the sentence of curriculum and deduplication brings to the work -- since this never further discussed. Note also that Figure 6 is never referenced in the main text, and that "Besides the dizziness due to conflicting ideas" on l30 is not adequate language for a paper.

Questions

Although exploring an interesting and valuable premise, this paper is insufficiently impactful and held back by its somewhat unclear and unprincipled methodology in the results section. I would recommend for the authors to: (1) unify their methodology across layer stacking/dropping/selective backpropagation and all models/datasets, to deliver a clearer results section; (2) improve the presentation of their results, with clearer figures (potentially plotting Pareto fronts as proposed in the main text); (3) get rid of the RST idea, which adds confusion for no reason. In its current form, I would rate this paper as a **Reject (3)**. **EDIT: following rebuttal I have updated my score to a Weak Accept (6)**. **Q1.** Based on the comments in W2 above on RST, could the authors clarify the value they see in RST? Especially regarding smoothing out fluctuations. **Q2.** Could the authors provide additional results for selective backpropagation at different compute budgets, and expand on the experiments combining SBP with different datasets?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors have included a limitations section.

Summary

The paper presents an analysis of 3 algorithms for training transformer models with a focus on efficiency. The authors present a way to measure wall clock time irrespective of the underlying hardware and make a principled comparison of the 3 algorithms by predefining a compute budget and adapting each algorithm and training recipe for the available time. The results of the paper show that none of the methods is universally an improvement over standard training.

Strengths

- Meta-analyses are very common in other fields but are sorely lacking in ML/Deep learning so the goal of this work is very welcome and in my humble opinion useful to the community. - Setting a compute budget and trying to optimize each algorithm accordingly is not as common as it should be in similar works.

Weaknesses

- The main weakness of the work is possibly the choice of algorithms to be analyzed. There are only 3 while there could be a plethora of others as mentioned in the paper's related work. Moreover, the 3 methods evaluated are not particularly widespread (possibly because they don't work as well as the paper shows). - Another possible weakness is the fact that there is separate hyper parameter tuning per method irrespective of the compute budget. This means that the results could be widely different were one to apply the above methods to a new problem where the optimal or near-optimal hyper parameters are not known a priori. - RST although fair, it is not a transferable metric which means that a different paper will not be able to compare with the RST results of this paper and will have to re-run the experiments to compute the relative speed of the methods on their hardware. This severely diminishes its usefulness.

Questions

My main reservation for the paper is whether the experiments are enough to prove useful to the community.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors properly present and analyze the limitations of their work.

Answer to rebuttal

First, I would like to note that I appreciate that the authors have written-up an extensive rebuttal to all reviewers, with significant additional results. The scope of the changes makes it difficult to evaluate the paper in its entirety again, but I won't blame the authors on this and take this as a quirk of NeurIPS allowing a 1-page results .pdf rebuttal for the first time in a while. The authors have adequately addressed some of my concerns: W1. (the added methods are interesting and paint a broader picture), W3., and likely W4. as well. However, I would maintain W2. (related to doubts around the introduced concept of RST). To better explain my concern, I simply think that a lot of lines are "wasted" on what is essentially a common idea of "comparing things that are comparable" / "rescaling". The authors highlights that the rescaling allows to compare across hardware and drivers version, but I seriously question why you would even consider doing this in the first place. This sounds like an overcomplexification of the evaluation setup employed. With that in mind, I will update my score to a **Weak Accept (6)** based on the extensive update proposed by the authors.

Thanks for your response

I read the author's responses carefully and believe that there is tremendous value in publishing these negative results, especially when they carry significant implications for future model training. I have raised my score to 5. However, I believe the review should focus on the original submission rather than newly added techniques. Thus I would not be able to further increase the score.

Thank you, concerns have been properly addressed.

The authors have addressed my concerns in a satisfactory manner. They have added three more recent efficient training algorithms to broaden the scope of the evaluation. They have also emphasized the ground-breaking nature of their observation that none of the popular recently proposed efficient training algorithms improves upon a learning-rate-decayed baseline. The authors have provided further clarification on the benefits of using Reference System Time (RST) as a measure of computational effort, addressing the concerns about fluctuations and variations in software optimization levels. They have also explained their definition of "significant improvement" and mentioned that the hyperparameters for each method were tuned independently. Additionally, the authors have provided more details on the methodology behind the "budget-adjusted baseline" and its implementation. Overall, my concerns have been properly addressed.