Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

**“The conclusion drawn from the paper is relatively superficial… ”** While we acknowledge the limitations of our study, we believe it offers valuable insights into a relatively unexplored area of the training process. Our findings highlight an important starting point that can inspire and guide future work. We hope this study demonstrates the value of pre-training checkpoints and encourages model builders to make them more widely available. To the best of our knowledge, no prior work has explored pre-training checkpoint experiments. Given the lack of alternative models to evaluate and the substantial resources we dedicated—over 1100 A100 GPU hours—we believe this work represents a significant step forward in this domain. We hope it underscores the feasibility of such research for the academic community, even within resource constraints. **“... and has been discussed in many previous works or some industry consensus”** We kindly request clarification or references to the specific previous work that the reviewer believes our conclusions overlap with. Without knowing the specific work the reviewer is referring to, it is challenging to address this concern or highlight the distinctions and contributions of our study effectively. **“The paper lacks some deeper insights into analyzing the parameter changes or loss changes during the pre-training or fine-tuning stages, which would provide theoretical support for the observed experimental phenomena.”** We believe our study still offers valuable insights. Throughout these analyses, we provide a meaningful examination of the training process that is often overlooked due to the current nature of training large language models. Moreover, our findings serve as an important starting point that can motivate and guide future work Loss changes have already been reported by the authors who released the models we used [1, 2]. We think our findings would complement the existing work that only reports loss changes. **“The paper's layout is somewhat chaotic, with some figures/tables and related text not on the same page, which poses a significant obstacle to reading.”** We believe these are easily addressable issues. If the reviewer can kindly point to the figures/tables mentioned, we can promptly make edits and upload a revision. **“In Section 5, the author claims that "the benefits of fine-tuning an LLM could exceed the benefits of continued pretraining", but in Section 7, the author also claims that "pre-training can improve models in unseen ways". These two viewpoints seem contradictory.”** These two findings are indeed not contradictory. Although pre-training can improve models in unseen ways, the improvement is not forever, there are diminishing returns. When the benefits of pre-training set plateaus, which can be identified by observing the datasets that show improvement in the early stage of training, it suggests that fine-tuning an LLM would exceed the benefits of continual pre-training. **“During the fine-tuning process, the paper conducts experiments on different specific tasks. What if it is in a general setting (such as AlpacaEval, MT-Bench), would the conclusions be different?”** Even though our datasets seem simple, the model does poorly on them during pre-training. Furthermore, we would like to clarify that our study's primary focus is the impact of supervised fine-tuning, not instruction following. The datasets (e.g., MT-Bench, Alpaca-Eval, Arena-Hard) are specifically designed with instructions, which are orthogonal to our core research questions. Instruction-heavy benchmarks introduce an additional confounding factor—namely, the model's instruction-following ability—rather than the core task-solving abilities we aim to study. That said, we agree that exploring the intersection of fine-tuning and instruction-following ability is an interesting direction for future work, and we hope our current findings can serve as a foundation for such analyses. [1] The Llama 3 Herd of Models [2] OLMo: Accelerating the Science of Language Models

**“ Verifying only one language model (OLMo-1B) is insufficient to provide convincing conclusions.”** Our study includes two models, OLMo-1b and Llama3-8B, and results on both models reach the same conclusions. Experiments that required pre-training checkpoints only include OLMo since Llama did not release checkpoints. We conducted an exhaustive search for pre-training checkpoints, including contacting several model authors. We are aware of only a few other models with checkpoints, which all have issues. 1) TinyLlama fixed a token problem mid-pre-training, which changed model behavior during pre-training. 2) RedPjama, which we experimented with extensively but performed poorly across all of our fine-tuning experiments. 3) Baichuan2, which is multi-lingual (introduces other issues) and is relatively unknown. 4) LLM360, which has a staged pre-training process that deviates from most other models. While we acknowledge the limitations of our study, we believe it offers valuable insights into a relatively unexplored area of the training process. Our findings highlight an important starting point that can inspire and guide future work. We hope this study demonstrates the value of pre-training checkpoints and encourages model builders to make them more widely available. To the best of our knowledge, no prior work has explored pre-training checkpoint experiments. Given the lack of alternative models to evaluate and the substantial resources we dedicated—over 1100 A100 GPU hours—we believe this work represents a significant step forward in this domain. We hope it underscores the feasibility of such research for the academic community, even within resource constraints. **“However, the practical issue is that if we want to train a specialized model, we should directly choose a well-pretrained model. We typically don't aim to start from the pretraining phase again.”** An LLM has to be pre-trained from scratch in order to become a “well-pretrained model”. By studying the effect of different amount of pre-training on resulting fine-tuning performance, we are hoping to provide insights on how pre-trained should be done. In addition, licensing restrictions on modern LLMs often prevent the selection of pre-trained models in some use cases. In such scenarios, companies frequently pre-train their own LLMs, making the insights from our study on fine-tuning and model optimization highly relevant for real-world applications.

**“The authors should provide a related work section to summarize the difference between this work and previous related studies.”** Section 2 Background (L88-L147) includes the related work section, in which we discuss a survey of previous work and its relationship with some of the most relevant prior works [1, 2]. **“The model backbone selected in this work is limited (only OLMo model). Have you tried other open-source models (e.g., OpenELM).”** Our study includes two models, OLMo-1b and Llama3-8B, and results on both models reach the same conclusions. Experiments that required pre-training checkpoints only include OLMo since Llama did not release checkpoints. We conducted an exhaustive search for pre-training checkpoints, including contacting several model authors. We are aware of only a few other models with checkpoints, which all have issues. 1) TinyLlama fixed a token problem mid-pre-training, which changed model behavior during pre-training. 2) RedPjama, which we experimented with extensively but performed poorly across all of our fine-tuning experiments. 3) Baichuan2, which is multi-lingual (introduces other issues) and is relatively unknown. 4) LLM360, which has a staged pre-training process that deviates from most other models. While we acknowledge the limitations of our study, we believe it offers valuable insights into a relatively unexplored area of the training process. Our findings highlight an important starting point that can inspire and guide future work. We hope this study demonstrates the value of pre-training checkpoints and encourages model builders to make them more widely available. To the best of our knowledge, no prior work has explored pre-training checkpoint experiments. Given the lack of alternative models to evaluate and the substantial resources we dedicated—over 1100 A100 GPU hours—we believe this work represents a significant step forward in this domain. We hope it underscores the feasibility of such research for the academic community, even within resource constraints. [1] Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs [2] Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer

**“The experiment employed only a single base model, which limits the generalization of the empirical findings.”** **“The parameters of the base model used in this paper amount to 1 billion, which does not include widely used model sizes of LLMs, such as 7 billion.”** **“In addition to the five candidate models mentioned by the authors, Baichuan2-7B may also be considered a candidate that has released intermediate checkpoints. https://huggingface.co/baichuan-inc/Baichuan2-7B-Intermediate-Checkpoints”** Our study includes two models, OLMo-1b and Llama3-8B, and results on both models reach the same conclusions. Experiments that required pre-training checkpoints only include OLMo since Llama did not release checkpoints. We conducted an exhaustive search for pre-training checkpoints, including contacting several model authors. We are aware of only a few other models with checkpoints, which all have issues. 1) TinyLlama fixed a token problem mid-pre-training, which changed model behavior during pre-training. 2) RedPjama, which we experimented with extensively but performed poorly across all of our fine-tuning experiments. 3) Baichuan2, which is multi-lingual (introduces other issues) and is relatively unknown. 4) LLM360, which has a staged pre-training process that deviates from most other models. While we acknowledge the limitations of our study, we believe it offers valuable insights into a relatively unexplored area of the training process. Our findings highlight an important starting point that can inspire and guide future work. We hope this study demonstrates the value of pre-training checkpoints and encourages model builders to make them more widely available. To the best of our knowledge, no prior work has explored pre-training checkpoint experiments. Given the lack of alternative models to evaluate and the substantial resources we dedicated—over 1100 A100 GPU hours—we believe this work represents a significant step forward in this domain. We hope it underscores the feasibility of such research for the academic community, even within resource constraints. **“The conclusions derived from the empirical analysis largely align with the established perspectives within this field, providing limited novelty.”** To the best of our knowledge, this is the first paper to study training dynamics using pre-training checkpoints. Please let us know which papers have conducted a similar analysis and where our conclusions have been previously published in the literature. **“There are no promising experiments demonstrating how these findings can inform the developing of LLMs.”** We believe that the question reviewer Reviewer S7Wz might be helpful in demonstrating the practical suggestions, so we pasted it here. **“Can you elaborate on potential signals or metrics during pre-training that could indicate an optimal point to stop pre-training and begin fine-tuning?”** Empirically, a practical approach is to use a set of validation datasets (for example, datasets in the first exp section) that have been examined to improve throughout pre-training. Those datasets do not require fine-tuning, but they can approximately entail the time when pre-training is sufficient. Once the performance on these validation sets plateaus or stops improving, it generally signals a diminishing return on continued pre-training. This could be used as a minimal bound to consider transitioning to the fine-tuning phase. **“how these findings can inform the developing of LLMs”** We appreciate the reviewer’s concern. However, the primary motivation of our study is not to propose immediate improvements to language model development, but rather to deepen our understanding of the effects of fine-tuning on model behavior. However, we have a few potential follow-up ideas in mind: 0. The answer to the question "Can you elaborate on potential signals or metrics during pre-training that could indicate an optimal point to stop pre-training and begin fine-tuning?" 1. Finding a balance point between the cost of training and the final performance. 2. Under a use case, achieve the best performance possible with an appropriate combination of pre-training and fine-tuning.

**Models and Experiments** Our study includes two models, OLMo-1b and Llama3-8B, and results on both models reach the same conclusions. Experiments that required pre-training checkpoints only include OLMo since Llama did not release checkpoints. We conducted an exhaustive search for pre-training checkpoints, including contacting several model authors. We are aware of only a few other models with checkpoints, which all have issues. 1) TinyLlama fixed a token problem mid-pre-training, which changed model behavior during pre-training. 2) RedPjama, which we experimented with extensively but performed poorly across all of our fine-tuning experiments. 3) Baichuan2, which is multi-lingual (introduces other issues) and is relatively unknown. 4) LLM360, which has a staged pre-training process that deviates from most other models. While we acknowledge the limitations of our study, we believe it offers valuable insights into a relatively unexplored area of the training process. Our findings highlight an important starting point that can inspire and guide future work. We hope this study demonstrates the value of pre-training checkpoints and encourages model builders to make them more widely available. To the best of our knowledge, no prior work has explored pre-training checkpoint experiments. Given the lack of alternative models to evaluate and the substantial resources we dedicated—over 1100 A100 GPU hours—we believe this work represents a significant step forward in this domain. We hope it underscores the feasibility of such research for the academic community, even within resource constraints. **“The benchmark datasets (flan-style) seem too simple and out of date for modern LLMs. For example, MT-bench, alpaca-eval, and arena-hard.”** Even though our datasets seem simple, the model does poorly on them during pre-training. Furthermore, our focus in on supervised fine-tuning, not instruction following. The datasets (e.g., MT-Bench, Alpaca-Eval, Arena-Hard) are specifically designed with instructions, which are orthogonal to our core research questions. Instruction-heavy benchmarks introduce an additional confounding factor—namely, the model's instruction-following ability—rather than the core task-solving abilities we aim to study. That said, we agree that exploring the intersection of fine-tuning and instruction-following ability is an interesting direction for future work, and we hope our current findings can serve as a foundation for such analyses. **Answer to reviewer’s questions:** _**“Could you provide more details on the selection criteria for the datasets and how they might influence the observed dichotomy between tasks learned during pre-training and those requiring fine-tuning?”**_ The datasets are selected based on potential data contamination. In addition to the datasets that we are sure were not contaminated, we also select the datasets and tasks (summarization, NLI, QA) that have been prevalent in NLP research. _**“How do you anticipate your findings would generalize to larger models or different architectures, given that your study was conducted on a relatively small model?”**_ As mentioned in Section xxx, the trend we found with Llama-8b is consistent with the findings with OLMo-1B. We believe the same finding is relatively generable for language models that are trained in a one-stage manner, which includes most Llama families, OLMo version 1, T5 family, etc. _**“Can you elaborate on potential signals or metrics during pre-training that could indicate an optimal point to stop pre-training and begin fine-tuning?”**_ Emprically, a practical approach is to use a set of validation datasets (for example, datasets in the first exp section) that have been examined to improve throughout pre-training. Those datasets do not require fine-tuning, but they can approximately entail the time when pre-training is sufficient. Once the performance on these validation sets plateaus or stops improving, it generally signals a diminishing return on continued pre-training. This could be used as a minimal bound to consider transitioning to the fine-tuning phase.