When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

Response to Reviewer oDvP

Thanks for your insightful comments! We corrected all typos in the updated version. **Re: is this the right form of the equation, or does this multiplicative scaling break down at larger parameter counts?** Understanding the reason for the bad extrapolation to 16B is valuable. We attempted different scaling formulations (both multiplicative and additive) and found that their extrapolation often fails at 16B mainly on En-Zh LLM. This inspires us that it is something else rather than the formulation causing this problem. After careful analysis, we discovered that the pretraining performance of the 16B En-Zh LLM can’t be well predicted by single-variable scaling laws, as shown in Figure 10 in the updated version: the actual performance is worse than the expectation. We argue that such mismatch caused by pretraining instabilities gets amplified during finetuning. This issue becomes more severe for PET as PET finetuning suffers more from unstable optimization. **Re: The authors had compared the scaling on even larger finetuning datasets to enable direct comparison to FMT in Figure 3.** While finetuning LLM with PET on larger-scale datasets (e.g. up to $1e6$~$1e7$ examples) is feasible theoretically, it is non-trivial in practice. This is because the hyperparameters of PET, including the learning rate, batch size and finetuning step, are highly biased and tuned for small-scale datasets to achieve the best quality but they can hardly generalize to larger settings. For example, we set batch size to 16 for PET, with which the model converges very slowly when using $1e6$ examples and may not converge at all due to the used finetuning step constraint. Readjusting these hyperparameters are required but are expensive and it may also break the scaling trend. **Re: Does the ordering here matter (e.g. FMT vs LoRA, or LoRA vs FMT) -- my understanding is that it should not.** Your understanding is correct: the ordering here doesn’t matter. **Re: How does zero-shot evaluation work in Figure 7? If the source language is unseen, how is the model able to get nontrivial performance on the task?** Although our LLMs are bilingual, they were pretrained on web-scale corpora which often unintentionally include data from other languages. In other words, the model has seen other source languages. We call it “zero-shot evaluation” because we never finetune the model on the “zero-shot” pairs but we expect that LLM is capable of generalizing its translation ability to them.

Response to Reviewer wU43

Thanks for your insightful comments! **Re: Including more tasks or languages, particularly low-resource translations, would enhance the paper's comprehensiveness.** We agree that working with different tasks and/or languages could enhance the generality and comprehensiveness of our study. That’s exactly why we performed experiments on two different language pairs (En-De and En-Zh) for machine translation as well as multilingual summarization. Adding more tasks should be beneficial, which we leave to the future given the timeline and compute resource constraints. **Re: it seems that the proposed scaling law may exhibit a significant mismatch for parameter-efficient fine-tuning when the model size is 16B.** When extrapolating to 16B, we noticed a high mismatch particularly for LoRA and Prompt on En-Zh LLM. After careful analysis, we argue that this problem stems more from the pretraining and finetuning rather than the proposed scaling law. Specifically, we find that the pretraining performance of the 16B En-Zh LLM is not well predicted according to a single-variable scaling law as shown in Figure 10 in the updated version. The 16B model performs slightly worse than the expectation, which we argue is caused by pretraining instabilities (such as job corruptions). Such mismatch is further amplified by the finetuning as PET finetuning suffers more from unstable optimization. **Re: How did the authors approach training with a decoder-only architecture for MT tasks? What prompts were utilized for MT training?** We perform finetuning for MT by converting MT examples into the LLM format and optimize the model with the conditional log-likelihood objective. More concretely, for each *source* and *target* example in MT, we concatenate them into a single sequence, i.e. *source* <sep> *target*. We feed this sequence to LLM and only compute log-likelihood on the *target*. We added this in the Appendix in the updated version. **Re: How might this differ from the scaling laws when using an encoder-decoder model?** Based on prior studies, the scaling of encoder-decoder models shows both similarities and differences compared to that of decoder-only models [1][2]. Exploring the scaling of finetuning encoder-decoder models is definitely an interesting research problem, which we leave to the future. >- [1] Ghorbani et al., 2021. Scaling laws for neural machine translation. >- [2] Bansal et al., 2022. Data scaling laws in NMT: The effect of noise and architecture.

Response to Reviewer R4DF

Thanks for your insightful comments! We corrected all typos and grammatical errors in the updated version. **Re: What model sizes are used in Figures 2/3/4?** In Figure 2, we used LLMs of parameters from 1B to 16B. In Figures 3 and 4, we mainly used 1B LLM. We added this information in the updated version. **Re: Figure 5 is unclear to me. Does the critical point refer to when A outperforms B in "A vs B"?** The critical point for “A vs. B” indicates the exact finetuning data size at which A performs equal to B given the different base model setup (x-axis) like model size or pretraining data size. We also explained this in the updated version. For example, if we investigate how the pretraining data size of the base 1B LLM would affect the decisions on finetuning data size, we can conclude from the plot (bottom-left) that when the pretraining data size is 2e11, the critical point for FMT vs LoRA at 50K indicates that we're more likely to achieve better performance with LoRA when we have data less than 50K while we have more headroom with FMT when we have more than 50K data.

Response to Reviewer gFLk

Thanks for your insightful comments! **Re: The core finding is that a multiplicative scaling model achieves better fits than a closely additive scaling model.** We appreciate your highlight of the proposed multiplicative scaling law. Note we consider it as a tool for analyzing the empirical results. Our main focus is to understand how LLM model size, pretraining data size, PET parameter size and finetuning data size affect the finetuning, which we achieve through this tool. We hope our follow-up findings could gain more attention, such as the importance of LLM model scaling, the ineffectiveness of PET scaling, and the high degree of task dependence of LLM finetuning. **Re: Why not add even more terms for even more expressivity? Can there be, and should there be, some controlling for the complexity of the law itself?** Expressivity is not the only consideration and we expect the law to be as simple/clean as possible. 1) There should be a trade off between expressivity and generalization. Adding more terms increases the expressivity theoretically but also brings in risks of overfitting, i.e. the model fits well on the given data but suffers from poor extrapolation. This problem becomes more concerning in LLM finetuning as the finetuning results can be very noisy, particularly for LoRA and Prompt tuning. 2) Keeping the formulation simple eases analysis. Note our analysis in Section 4 is mainly based on the comparison of different scaling exponents. But such direct comparison is significant only for the proposed multiplicative formulation without additional terms. In the additive formulation, for example, comparing $\alpha$ with $\beta$ is complicated due to the difference of $A$ and $B$. **Re: Where are the counterparts of Table 2 for the other tasks and other metrics (besides perplexity)?** For the fitting errors on WMT En-Zh and MLSum, we add them in Table 6, Appendix in the updated version. We noticed that the multiplicative scaling law performs worse on MLSum compared to the additive one; but overall, our proposal generalizes slightly better over different tasks and finetuning methods. For the scaling laws on metrics other than PPL, we didn’t explore them. Firstly, these metrics highly correlate with PPL as demonstrated by the high Pearson’s $r$ in Table 7, Appendix in the updated version. Secondly, adopting PPL for scaling law is the standard practice in the literature [1,2,3] because improvements in PPL may not be well reflected in other metrics. Take the binary classification task as an example. Improving the probability (thus PPL) of the correct answer from 0.1 to 0.2 doesn’t change the classification accuracy (take 0.5 as the threshold). **Re: BLEURT-RougeL picture "shows high correlation with the PPL scores in general"** We performed Pearson correlation analysis between PPL and BLEURT/RougeL and added Table 7 to Appendix in the updated version. Most Pearson’s $r$’s absolute value is larger than 0.9, and all of them are significant at $p<0.01$ except FMT for LLM model scaling on WMT En-De. **Re: This guidance is very familiar and doesn't need to be characterized with a "scaling law". Is there more specific guidance implicit in this work?** While the findings align with intuition and look familiar, the scaling law is still necessary as it offers a quantitative measure for the impact of different scaling factors on the finetuning. This enables the comparison between scaling factors and gives readers more convincing evidence beyond intuition and experience. We consider these findings reached through large-scale and systematic experiments and characterized by our scaling laws as significant contributions to the community. We need to emphasize that our results provide rich information and the guidance is multi-dimensional, not just specific to finetuning. For example, finetuning benefits more from LLM model scaling than pretraining data scaling, which suggests a different recipe compared to the computation optimal scaling for the pretraining. In terms of finetuning, while PPL generally goes down with more data, the expected return from collecting more finetuning data differs greatly across finetuning methods as demonstrated by the scaling laws, and PET shouldn’t be expected to perform well with small LLMs. In the era of LLM, our study represents itself as the first kind to explore the model, data and finetuning method for the scaling of LLM finetuning, which we believe contains intriguing and substantial contents to the community. We hope the reviewer could reconsider our paper. >- [1] Kaplan et al., 2020. Scaling Laws for Neural Language Models >- [2] Ghorbani et al., 2021. Scaling Laws for Neural Machine Translation >- [3] Hoffmann et al., 2022. Training Compute-Optimal Large Language Models