General response
Thanks for your insightful comment.
General response:
Many reviewers are interested in performance for different LLM and different datasets and question our conclusion because of the pre-training losses come from different corpora. We have uploaded a new version with new experiments. We use a random sample of the Pile test corpus to align the pre-train losses of LLaMA, LLaMA2, and Pythia series. We conduct SFT and RFT experiments on GSM8K benchmark with Pythia series. We conduct SFT experiments on MATH benchmark with LLaMA and LLaMA2. Our detailed results are listed in the updated PDF of section H in appendix. I will summary our main findings here.
(a) The pre-training losses are still negatively linear correlated to SFT performance including Pythia series models.
(b) RFT improves performances of Pythia series models significantly. (Pythia-410M SFT 5.6 vs RFT 18.9, Pythia-2.8B SFT 18.8 vs RFT 34.6).
(c) For MATH benchmark, LLaMA series models still improving performance linearly when double fine-tuning data.
W1 & Q3: We have added Pythia on GSM8K and LLaMA on MATH experiments.
W2: In Appendix D, we have covered the RFT inference and training FLOPs, which have a significantly lower computational requirement (1 \times 10^{-4}) compared to pre-training. This level of computational cost is considered acceptable. Although some may argue that using GPT4 API for data augmentation is less expensive, this neglects the computational cost associated with inferencing using GPT4. Additionally, our approach solely relies on models like LLaMA that we aim to enhance, without the need for GPT4. In contrast, other math SFT papers like WizardMATH, Meta-Math, and Mammoth require GPT API for data augmentation.
W3: We are also interested in investigating the relationship between reasoning performance with human-written fine-tuning data amount and model-augmented fine-tuning data amount which is not be researched previously. We also find the distinct reasoning paths are the most important factor for augmentation dates.
Q1: We have conducted experiments on human alignment experiment. We fine-tune LLaMA-7/13/33B models on ShareGPT dataset with (1, 1/4, 1/16, 1/64) amount of data and use MT-Bench to measure the alignment performance. For LLaMA-7B models, we get MT-Bench score with 5.88, 5.85, 5.61, 5.11. For LLaMA-13B models, we get MT-Bench score with 6.13, 6.03, 5.66, 5.24. For LLaMA-33B models, we get MT-Bench score with 6.63, 6.66, 6.17, 5.99. We find that model performances improves with data mount increases in the most scenarios.
Q2: We find that when the pre-trained models get larger (like LLaMA-65B (+0.4) or LLaMA2-70B (+1.6)), the RFT improvement is insignificant. We consider the main reason is larger models have gained considerable reasoning ability during pre-training, and the space for improvement gets smaller for them.