Two-stage LLM Fine-tuning with Less Specialization and More Generalization

We thank the reviewer for valuable suggestions and comments. We address your questions below in detail. **Regarding experiments on finetuning on more tasks.** Sorry for the confusion. In Table 10 of the appendix, we have shown experimental results on two additional finetuning datasets including openbookQA, which is a QA dataset and SNLI as another NLI dataset. Due to space limit, we put these results in Appendix,. The results are consistent with our main experiments in the paper. Also based on our gradient analysis given in section 3.2-3.3, we can see that format specialization happens for finetuning tasks with any well-defined output syntax. The reason is that the gradient associated with such syntax is present consistently in all finetuning data points, resulting in a strong gradient signal. Lots of fine tuning tasks in the NLP community today contain such syntax elements. We would deeply appreciate it if the reviewer finds that these clarifications are helpful and strengthens our manuscript, and raise the score accordingly.

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes tomorrow. We appreciate the feedback, and will appreciate it if you would like to comment on our rebuttal and additional results and revise the score accordingly. Thank you!

Thank you to the authors for their rebuttal. I hope the authors will consider including additional experiments in the main paper in their next revision. After reading comments from other reviewers, my confidence in the quality of the work has increased, leading me to believe that it is sufficiently robust for acceptance. Consequently, I have raised my score to 8.

We thank the reviewer for valuable suggestions and comments. We address your questions below in detail. **Regarding experiments on more models (different base models including decoder-only model).** We have done additional experiments on different base models beyond mT5 shown in the main paper, including T5XXL, T5XL and PaLM-8B. Due to limited space in the main paper, we put these results in Table 11, 12, 13 in Appendix. The PaLM-8B model we used in the additional single-task experiments is a decoder-only model. We report the results in Table 13 in the Appendix. As we can see from these results, the additional results are consistent with our main experiments on the mT5 XXL model, where ProMoT outperforms prompt-tuning in-domain and outperforms standard finetuning out-of-domain. In particular, as an improved finetuning method, ProMoT shows significantly less forgetting of in-context capabilities compared to standard finetuning and may even improve over out-of-domain but semantically related tasks compared to the pretrained model.

We thank the reviewer for valuable suggestions and comments. We address your questions below in detail. **Regarding experiments on more models.** Sorry for the confusion. We have done additional experiments on other models beyond mT5, including T5XXL, T5XL and PaLM-8B. Due to limited space in the main paper, we put these results in Table 11, 12, 13 in Appendix. As we can see from these tables, results on these additional models are consistent with our main experiments on the mT5 XXL model, where ProMoT outperforms prompt-tuning in-domain and outperforms standard finetuning out-of-domain. In particular, as an improved finetuning method, ProMoT shows significantly less forgetting of in-context capabilities compared to standard finetuning and may even improve over out-of-domain but semantically related tasks compared to the pretrained model. **Regarding experiments on more datasets.** We have included experiments on two additional finetuning datasets (SNLI and OpenBookQA) in Appendix Table 10 for the mT5 XXL model. The results are consistent with our main experiments in the paper. **Regarding highlighting similarity tasks.** Thanks for the suggestions. We have put some texts in the tables’ captions to highlight the improvement over the similar tasks.

We thank the reviewer for valuable suggestions and comments. We address your questions below in detail with additional experiments and in-depth analysis. **Regarding practical utility of ProMoT.** Thanks for the comments. We agree with the reviewer’s insight that the most significant application of ProMoT would be in domains where high quality finetuning data is limited. In such scenarios, it is imperative to have methods that are 1) robust against severe forgetting due to finetuning data homogeneity, which will cause format specialization as we extensively demonstrated in this work. 2) can maximize the value of each datapoint through improved generalization. We would like to stress that this kind of situation is not a small niche, especially when language models are applied to new and complex domains where it is extremely difficult to collect high quality supervised finetuning data even for one task - autonomous LLM agents is one such area amongst many. Given this successful proof of concept, we believe that our method will open up new possibilities in these domains and will be of interest to a broad community of ICLR researchers. **Regarding massive multi-task experiments.** Thanks for the comments. As discussed above, the focus of this work is more towards low data / high generalization requirement regime. With our best effort given computational resources constraints, we choose not to run massive multi-task training on large-scale datasets (such as the FLAN mixture, among others). However, as demonstrated by a recent work (Table 1 in https://openreview.net/pdf?id=VrHiF2hsrm), forgetting of in-context learning capabilities also exists in the massive multi-task regime, where methods like ProMoT could be helpful. We hope to explore the massive multi-task regime with ProMoT as a future work, but we do believe that our current results are self-contained, thorough and significant enough to be known by the community. **Regarding 4-shot evaluation results.** Thanks for the suggestions. We have added the 4-shot evaluation results for fine-tuning v.s. ProMoT with multi-task training in the table below. As we can see from the table, the results are consistent with the 1-shot result that ProMoT can produce a generally better model on these evaluation tasks with 4-shots prompts. Note that as True/False format has been seen during fine-tuning, the model with standard fine-tuning can also do well on CB and WiC tasks. We have added this additional result to the Appendix. | | Pretrained Model - 4-shots | Multi-task Finetune - 4-shots | Multi-task ProMoT - 4-shots | |---------------|----------------------------|-------------------------------|-----------------------------| | cb | 51.79 | 82.14 | 78.57 | | wic | 49.69 | 52.82 | 52.5 | | TriviaQA | 19.02 | 17.75 | 22.26 | | Web_questions | 13.04 | 12.25 | 12.5 | | WMT16 En-De | 8.83 | 0.24 | 18.84 | | WMT16 En-Ro | 3.92 | 0.54 | 18.81 | | XSum | 2.35 | 0.34 | 5.04 | | WikiLingua/en | 1.33 | 0.39 | 1.92 | **Regarding the Prefix + Parameter tuning baseline.** Thanks for the comments. We have included a ‘joint fine-tuning’ setting in the ablation study (Table 6), which trains the soft prompt and the main model jointly exactly as the comments suggested. In evaluation, we only use the main model and remove the soft prompt on evaluation tasks with different formats. As we can see from the table, the prefix + parameter tuning joint training still suffers from significant loss of general in-context learning performance, as the format learning can not be separated into the prompt if we train the two parts jointly. **Regarding the prefix-tuning column.** Thank you for your suggestion on the table organization. For the source tasks, we report the in-domain performance of prompt-tuning in Table 2, showing that prompt-tuning is indeed inferior compared to standard finetuning or ProMoT in-domain. For evaluation on other tasks, exactly as the reviewer mentioned, since prompt tuning does not touch the parameters of the pretrained model, it has the same few-shot performance on evaluation tasks as the pretrain model where we remove the soft prompt. We have also made this clear to the audience in the footnotes of Table 3 and 4. We hope these would clarify the prompt tuning results well.

**Regarding evaluating prefix-tuning on other tasks.** Thanks for the suggestions. As requested by the reviewer, we evaluated the performance of using the prompt trained from the English to French translation task WMT14 En-Fr using prompt tuning on other translation tasks with different language pairs. We show some examples below, we add a 1-shot prompt before the actual input in evaluation denoted by [1-shot prompt]. As we can see from the table, when we use the prompt from WMT14 En-Fr to test on En-De and En-Ro translation tasks, the predictions are still in French, which implies that the format information of outputting in French is learned by the prompt. | Input | Target | Prediction with the soft prompt trained from En-Fr | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------| | [1-shot example omitted…] Translate English to German: The first series of The Lenny Henry Show aired in 1984, and in the Nineties he was known as, among other things, chef Gareth Blacklock in the comedy series Chef! | Die erste Reihe der Lenny Henry Show ging 1984 auf Sendung, und in den Neunzigern war er unter anderem als der K\u00fcchenchef Gareth Blacklock in der Comedyserie Chef! bekannt. | Nous souhaitons aider nos clients \u00e0 r\u00e9ussir. | | [1-shot example omitted…] Translate English to German: I know Georgy won't go. | Ich wei\u00df, dass Georgij nicht hingehen wird. | Cela ne signifie pas une Europe minimaliste \u2013 et je veux | | Input | Target | Prediction with the soft prompt trained from En-Fr | |-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------| | [1-shot example omitted…] Translate English to Romanian: People using ad-blocking software are likely to run into a few hitches. | Persoanele care folosesc programe software de blocare a reclamelor \u00eent\u00e2mpin\u0103 c\u00e2teva constr\u00e2ngeri. | Les personnes qui utilisent des logiciels de blocage des annonces | | [1-shot example omitted…] Translate English to Romanian: Brent futures were up 61 cents Wednesday at $48.36 per barrel. | Cot\u0103rile Brent au crescut miercuri cu 61 de cen\u021bi, ajung\u00e2nd la 48,36 USD per baril. | Les futures sur le Brent ont gagn\u00e9 61 centimes mercredi | **Regarding the performance on PaLM.** Thanks for the comments. We would like to highlight that ProMoT, as an improved finetuning method, still vastly outperformed prompt-tuning in-domain and significantly out-performed standard finetuning out-of-domain, providing a very attractive trade-off point between the two extremes. However, we acknowledge that our results on PaLM models are somewhat more exploratory, which is why they are placed in the appendix. Factors such as the effectiveness of prompt tuning on these models will affect their ability to absorb the format part of the finetuning task. This is an interesting topic that warrants a dedicated examination in a future work. We would deeply appreciate it if the reviewer finds that these additional experiments and clarification are helpful and strengthens our manuscript, and raise the score accordingly.

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes tomorrow. We appreciate the feedback, and will appreciate it if you would like to read our rebuttal and additional results, and revise the score accordingly. Thank you!