GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length

Response to Reviewer gxVT

Dear Reviewer gxVT, We sincerely thank you for your time and effort in reviewing our paper. We value your suggestions and address them point by point in the following: **Q.1. The paper lacks comparison to the BERT [1] pre-training, which used a two-step pre-trianing approach with a growing context window length.** The pre-training methods of BERT and GPT-NeoX are distinct. BERT employs a two-step approach with an expanding context window, focusing on masked language modeling and next sentence prediction. In contrast, GPT-NeoX utilizes an autoregressive training strategy. Considering these differences, we did not choose the BERT pre-training as a baseline. **Q.2. The figures are confusing with different arrows pointing at the lines with same colors (Figure 3 & 5).** Thanks for your suggestion, we modified the figure to make it clearer. **Q.3. The experiments are conducted on a single neural architecture and the provided architecture sizes are considerably small compared to existing LLMs.....How would the GrowLength method work with substantially larger model 7B+?** Larger models, such as those with 7B+ parameters, require significantly more computational resources. Implementing the GrowLength method at such scales could be prohibitively resource-intensive in terms of computational power and energy consumption. This limitation could affect the practicality of using the method in real-world applications, where resource constraints are a significant consideration. Moreover, it is common practice to use smaller models to verify experimental settings, as seen in the Llama2 paper. In this paper, the GQA setting is verified using a smaller model (30B) before being applied to a larger model (70B). Also, it has been found that much smaller models' performance can predict the performance of huge models. This has been discussed in GPT-4 technical report[1], PaLM-2 report[2] and Chinchila[3]. For examples, by the figure 1 and figure 2 from GPT-4's report, much smaller models (1000x smaller) can be used to predict GPT-4's performance. Hence, we kindly argue that, the lack of results from pretty large models does not harm the soundness of our mehtods. [1][GPT-4 Technical Report](https://arxiv.org/pdf/2303.08774.pdf) [2][PaLM 2 Technical Report](https://arxiv.org/pdf/2305.10403.pdf) [3][Training Compute-Optimal Large Language Models(Chinchilla)](https://arxiv.org/pdf/2203.15556.pdf) **Q.4. Where are GrowLength-1 and GrowLength-2 defined?** In Figure 4, 'GrowLength-1' represents the model trained using our growth strategy. In contrast, 'GrowLength-2' employs the same strategy but processes approximately twice as many tokens as 'GrowLength-1'. 'GrowLength-1' demonstrates accelerated training times compared to models using a fixed context length of 1024, as shown in Figure 2. When processing more tokens, the training duration of 'GrowLength-2' is comparable to that of a fixed-length model, namely 'LLM1024'. We have included both models in the figure to ensure a comprehensive and fair comparison. **Q.5. How would the GrowLength method work with substantially context windows?** **if we understand correctly, the reviewer want the performance of LLM with longer context windows** The LLM pre-trained by our proposed method consistently demonstrates superiority over all the baselines when comparing GrowLength-1, LLM1024, and LLM128. The results are presented in Figure 4. Your feedback is invaluable to us, and we trust that our response has effectively addressed your concerns. In light of this, we would be grateful if the reviewer could reassess our work and possibly reconsider the score given. Thanks,\ Authors

[1/2] Response to Reviewer w2Jc

Dear Reviewer w2Jc, We sincerely thank you for your time and effort in reviewing our paper. We value your suggestions and address them point by point in the following: **Q1: How to determine the growing context window size during training (like 128, 256, 512, …) is not rigorously studies. Will this exponential growth be too fast, especially for the latter stage, or even longer sequence (e.g., 1B tokens)? And maybe direct positional extrapolation will not work well when growing too fast.** Our growth schedule was determined empirically. We agree that significant discrepancies in stage lengths could impair performance during pretraining, as indicated in Figure 5. However, our experiments suggest that, at least in our current setting, exponential growth does not negatively affect our results. Regarding super-large context window sizes, the proposed methods may not be effective. Some existing studies [1,2] have extended the context length by fine-tuning, for example, scaling Llama-2 from 4k to 8k or even 32k. This supports the viability of our method. Given the abundance of data in the pretraining phase compared to fine-tuning, we believe that a growth strategy could be effective even with larger length variations between stages. It is also important to note that we do not claim our schedule is optimal. The key innovation of our research is the introduction of increasing context lengths during pretraining, which we believe enhances the process. [1][Extending Context Window of Large Language Models via Positional Interpolation](https://arxiv.org/abs/2306.15595) [2][LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307) **Q2:The unique challenge when applying content window extension to pretraining stage is not very clear. It seems that simply using the technique proposed for fine-tuning also works well.** Our approach aims at reducing pretraining time for models with a fixed sequence length, such as 1024 tokens. While our method focuses on minimizing pretraining duration, we recognize that our fine-tuning technique is still applicable and beneficial for various purposes during the fine-tuning stage. **Q3: No final testing results are presented. Although this paper is working on optimization but not generalization. But final performance on testing set of various tasks is the thing that matters a lot. Besides loss, these results should also be included and analyzed.** We conducted experiments to investigate the performance comparsion on other downstream tasks. Specifically, we ran several tasks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master). The results show that our proposed method outperforms the baseline in 7 out of 9 tasks. | Dataset | Staged Training | Fixed Length Training(1024) | |---------|---------|---------| | boolq | **0.4465** | 0.3804 | | multirc | **0.0189** | 0.0105 | | rte | 0.5126 | **0.5235** | | wic | **0.5235** | 0.5047 | | wnli | **0.4507** | 0.4366 | | cb | **0.5179** | 0.4107 | | qasper | **0.3356** | 0.0 | | webqs | **0.0009** | 0.0005 | | wsc | 0.3654 | 0.3654 | **Q4: Will this method be sensitive to the length distribution of the training data? If the pre-training dataset has few samples with short sequences, will GrowLength still be effective?** Our GrowLength method remains effective even when the pre-training dataset contains a limited number of short sequences. Additionally, the nature of pre-training data for Large Language Models (LLMs) typically involves large volumes, allowing for the selective inclusion of pre-training text data. It is important to highlight that our proposed pre-training approach demonstrates superior generalization capabilities for longer texts, as illustrated in Figure 4. This indicates that our method is robust against variations in test text length.

Response to Reviewer jdbk

Dear Reviewer jdbk, We sincerely thank you for your time and effort in reviewing our paper. We value your suggestions and address them point by point in the following: **Q1: My largest concern lies in the validation of the experiment. The model is trained around 20000s (5.56h) at most, which is too short to validate the pretraining process since the model is far away from convergence. Notice that [1] trains model for 300B tokens, and a 160M LLM usually needs several days to converge on 16 V100s, to my knowledge. I completely understand that resource demand is high for researchers; however, it is hard for me to agree with the conclusion from the present experiment.** Thank you for your question and feedback. We conducted an experiment to continue pre-training the current models, as illustrated in Figure 1. As you pointed out, 'a 160M LLM usually requires several days to converge on 16 V100s,' we have decided to reserve the pretraining of larger models for future work. We are confident that our experiments, in their current form, have already substantiated that our proposed method can significantly reduce the time required for pretraining. **Q2: A shorter sequence surely leads to a smaller computation complexity. Therefore, this insight is not really original.** Thank you for your observation. While it's true that shorter sequences reduce computational complexity, our experiment focused on the efficiency of training without compromising the model's quality. The key contribution of our work is not just identifying the reduction in computation due to shorter sequences, but also demonstrating how our specific methodology effectively maintains, or even enhances, the model's quality within these constraints. This aspect of our research extends beyond the straightforward observation of reduced computational complexity, offering a more nuanced and practical approach to model training. Your feedback is invaluable to us, and we trust that our response has effectively addressed your concerns. In light of this, we would be grateful if the reviewer could reassess our work and possibly reconsider the score given. Thanks,\ Authors

Response to Reviewer TKo2

Dear Reviewer TKo2, We sincerely thank you for your time and effort in reviewing our paper. We value your suggestions and address them point by point in the following: **Q(a) How do the authors transition between stages? Do the authors use position interpolation or extrapolation to provide a smooth transition when the sequence length increases? Furthermore, a comparison study to a different position embedding would highlight the importance of ROPE embeddings for a smooth transition across the stages.** We thank you for the insightful and valuable question. Regarding the stage transition, through the experimental results in Figures 1, 2, and 3, we found that directly using position extrapolation makes the transition smooth enough. For different position embeddings, we conducted the following experiments to investigate the impact of position embedding. These include various position embeddings, such as absolute positional embedding (GPT-2), and Alibi positional embedding (BLOOM). **Q(b) How many sequence batches do the authors use for each stage of training? Is it proportionally set to the sequence length at each stage?** We thank you for your insightful comment and would like to clarify the experimental setting. We maintained a constant product of batch_size*seq_length for all sequence lengths. Specifically, we used a batch size of 12 for a sequence length of 1024, 24 for 512, 48 for 256, and 96 for 128. **Q(c) Related to my second question, if the batch sizes have been changed at each stage, have the hyperparameters (Learning rate, batch size, etc.) of the baseline experiments been optimally tuned for fair comparisons?** Thank you for your question. We would like to explain the settings for hyperparameter tuning, such as learning rate and batch size. In large language model (LM) pretraining, it's common practice not to adjust hyperparameters, including learning rate, extensively. This is due to the unaffordable training costs. In line with this standard practice, we did not specifically tune these hyperparameters for each stage. However, we ensured that the comparisons remained as fair and consistent as possible within the constraints of our experimental setup. **Q(d) How does the improvement in perplexity relate to improvements in downstream performance? Any fine-tuned or zero-shot performance will show the general efficacy of the proposed method.** Thank you for your insightful comments regarding the relationship between perplexity improvements and downstream task performance. We appreciate the opportunity to further clarify and elaborate on this aspect of our work. You rightly point out the complexity in correlating perplexity improvements directly with downstream task performance. We agree that perplexity, while a valuable measure of a model's predictive capabilities, is not the sole indicator of performance in practical applications. We conducted experiments to investigate the performance comparsion on other downstream tasks. Specifically, we ran several tasks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master). The results show that our proposed method outperforms the baseline in 7 out of 9 tasks. | Dataset | Staged Training | Fixed Length Training(1024) | |---------|---------|---------| | boolq | **0.4465** | 0.3804 | | multirc | **0.0189** | 0.0105 | | rte | 0.5126 | **0.5235** | | wic | **0.5235** | 0.5047 | | wnli | **0.4507** | 0.4366 | | cb | **0.5179** | 0.4107 | | qasper | **0.3356** | 0.0 | | webqs | **0.0009** | 0.0005 | | wsc | 0.3654 | 0.3654 | Your feedback is invaluable to us, and we trust that our response has effectively addressed your concerns. In light of this, we would be grateful if the reviewer could reassess our work and possibly reconsider the score given. Thanks,\ Authors