Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance

Response to Reviewer FeNh

We thank the reviewer for appreciating our empirical work and for raising insightful questions. We respond to all points below. **Weakness I**: We acknowledge the reviewer's point about the lack of theoretical insights. While there is substantial empirical evidence demonstrating the effectiveness of multi-task learning, modern theoretical understanding remains limited [1]. Although earlier works provide strong theoretical frameworks for multi-task learning [2,3], extending these frameworks to modern deep learning methods presents significant challenges. To address this point, we have added relevant citations to the manuscript, along with a paragraph in the limitations section. [1]: Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint, arXiv:2009.09796,2020. [2]: Theodoros Evgeniou and Massimiliano Pontil.Regularized multi–task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 109–117,2004. [3]: Carlo Ciliberto, Youssef Mroueh, Tomaso Poggio, and Lorenzo Rosasco. Convex learning of multiple tasks and their structure. In International Conference on Machine Learning, pp.1548–1557. PMLR,2015. **Weakness II**: Regarding the use of adapters and prompt-tuning, we agree with the reviewers' suggestion that exploring additional fine-tuning techniques could provide valuable insights. However, such experiments fall outside the scope of this work. Our study specifically examines different data mixes, isolating this variable to analyze its effects comprehensively. While we anticipate similar findings given the strong dependency of fine-tuned models on their training data, investigating the impact of alternative fine-tuning techniques beyond standard training of the full model and all its weights would be an interesting direction for future research. Finally, we thank the reviewer for bringing the paper by Liu et al. to our attention. While their approach is certainly related to ours, there is a major difference: rather than creating a single model capable of solving multiple tasks, they share the earlier layers of the model across tasks while using separate 'heads' for each task. This approach is understandable given the limitations of BERT-related architectures. With more modern and powerful architectures, these separate 'heads' are no longer necessary. Nonetheless, the work by Liu et al. represents an important early contribution to multitask fine-tuning, and we have added a citation to acknowledge it.

Response to Reviewer KnZt

We thank the reviewer for appreciating our presentation and for their insightful comments. We address all points below. **Weakness I**: First, regarding the mention of '200+ models,' we acknowledge that this terminology may have been unclear without additional context. In this work, we refer to the checkpoint produced by each training experiment as a 'model.' However, we only use four base models, as described in Section 4.1: Phi-3-small, Phi-3-mini, Llama-3-8B, and Mistral-7B-v3. Thus, the 200+ models represent fine-tuned versions of these four base models. Based on a similar comment from Reviewer d8tj, we have added a clear explanation of the number of distinct training experiments that resulted in these fine-tuned models. For completeness, we repeat our response to Reviewer d8tj here: > Specifically, for each base LLM, we conducted 55 training experiments, and comparisons are made only within this subset. These 55 experiments per base LLM arise from the nine datasets, following the process outlined in Section 2.2. As shown, the number of training runs is calculated as $\binom{n}{1} + \binom{n}{2} + \binom{n-1}{1} + \binom{n}{n} = 55$, where $n=9$. We have added a concise explanation of this calculation to Section 4.1 in the manuscript. **Weakness II**: Regarding the reviewer's concern about the dataset variety: we acknowledge that the financial domain is vast, encompassing many intricate details that are not fully captured in our selection of seven core training datasets. However, we believe that these seven datasets provide a robust representation of critical financial tasks, including sentiment analysis, financial classification, named entity recognition, conversational capabilities, and tabular and mathematical understanding. These tasks collectively cover a substantial subset of financial NLP challenges. We also want to highlight the computational considerations inherent in this empirical study. Adding more datasets would significantly increase the computational demands due to the combinatorial growth in the number of training runs (as visualized in Figure 2). Given this trade-off, we carefully selected seven datasets that balance computational feasibility with a representative scope of financial tasks. Additionally, we have added a new limitations subsection to the manuscript to explicitly discuss this point. Regarding novelty and comprehensiveness, we agree that the work by Xie et al. is indeed relevant, and we note that our study shares some datasets with theirs. We have added a citation to their work in acknowledgment. However, our research makes a unique contribution by systematically evaluating the effects of dataset combinations on fine-tuning performance. Unlike prior studies, which often focus on fine-tuning a single model or evaluating pre-trained models, our work explores how the choice of training data mix can influence downstream performance. This emphasis on data selection as a critical variable distinguishes our study and offers valuable insights into fine-tuning strategies, particularly for financial NLP tasks, and potentially for the broader NLP community. **Weakness III**: We thank the reviewer for suggesting the inclusion of error bars. In response, we have added margin of error values to Table 2, as recommended, to highlight significance. Additionally, we took the opportunity to enhance Figure 1 by including error bars for greater clarity.

Response to Reviewer d8tJ

We thank the reviewer for raising valuable points to improve our work. We address all points below. **Open source**: In response to the reviewer's recommendation, we have open-sourced our project, including the source code and all datasets, accessible at https://anonymous.4open.science/r/cocktail_effect-54F8/README.md. While it is not feasible to upload all 200 saved models due to size limitations, the provided code enables full reproduction of our experiments or any chosen subset. Also, we'd be happy so share our trained models with researchers by uploading a subset of the trained models, upon request. **Weakness I**: Regarding the first weakness described, we appreciate the reviewer's focus on ensuring a fair comparison between the trained models, as this is indeed a key consideration. We assert that our comparison is fair for several reasons: - First, we clarify that we are not comparing a single model to a 'best-of-200' model. We recognize that our original language may have been unclear, so we have adjusted the wording in the paper to improve clarity. Specifically, for each base LLM, we conducted 55 training experiments, and comparisons are made only within this subset. These 55 experiments per base LLM arise from the nine datasets, following the process outlined in Section 2.2. As shown, the number of training runs is calculated as $\binom{n}{1} + \binom{n}{2} + \binom{n-1}{1} + \binom{n}{n} = 55$, where $n=9$. We have added a concise explanation of this calculation to Section 4.1 in the manuscript. - Second, regarding hyperparameter selection, we tested several configurations to find one that achieved good convergence across the various training runs. Given this hyperparameter search, additional tuning would likely not significantly impact the results, as this process has already been conducted. However, due to the large scale of experiments in this paper, a truly exhaustive search was not feasible, and we have added this as a limitation. - Lastly, our primary goal was to identify the optimal dataset mix, making it essential to train models on all possible combinations. **Weakness II**: Regarding the second weakness the reviewer has raised, we would like to thank the reviewer for pointing out these relevant papers, which we have now added acknowledgments to in our paper. We would also like to add that besides the insight the reviewer has mentioned, namely that using general instruction datasets and/or mathematical reasoning datasets is greatly helpful, we generally do not get to pick which tasks to include. The tasks are a given target set, based on the specific domain. Once this set has been established, the task of finding the optimal subset to train on (per task) is similar to any other hyperparameter search, where empirical experiments are necessary. Tables 5,6,7 in the appendix demonstrate the often non-intuitive nature of optimal combinations. **Question I**: The reviewer raises an important consideration regarding format adherence. To address this concern, we conducted a detailed empirical evaluation of the instruction-following capabilities of GPT-4-o within our benchmarks. Specifically, we identified all instances where our rule-based evaluation failed to extract an answer, which directly corresponds to cases where GPT-4-o did not correctly follow the required format. The results, presented in Table 1 below, show that the number of formatting issues is negligible, thus reducing the need to use LLM-as-a-judge in this case. Furthermore, this demonstrates that the gains of the smaller fine-tuned models are due to improved performance relative to GPT-4-o, rather than being limited to better format adherence. **Table 1: Number of GPT-4-o formatting errors per task, presented as a fraction (errors/total test set samples).** | **Dataset** | **# GPT-4-o Formatting Mistakes** | |--------------------|-----------------------------------| | Twitter SA | 3/2,388 | | TwitterTopics | 2/4,117 | | FinNerCLS | 1/3,502 | | FPB | 0/970 | | ConvFinQA | 0/1,486 | | FinQA | 0/1,125 | | Headline | 0/20,547 |