Can Models Learn Skill Composition from Examples?

Summary

The paper studies the extent to which LLMs can learn compositional generalization over skills by finetuning with suitable training examples. Based on abundant experimental results, the paper concludes that finetuning using a dataset with skill composition examples helps the model build a “meta-skill” that allows them to generalize to more complex (i.e., composition of more skills) tasks. The paper also finds that the training samples containing higher-order skill compositions are more efficient in eliciting such a meta-skill. The paper is quite easy to follow and the conclusions might inspire more interesting applications on LLM’s finetuning.

Strengths

- The paper proposes a dataset that consists of examples with different selected skills for the analysis of skill composition. - The results in this paper show that finetuning not only provides new knowledge to the LLM but also the skill of composing different skills, which potentially inspire novel applications using finetuning. - The proposed method (although looks simple) efficiently improves the model’s performance on skill composition. The observation that the “skill-richer” data can induce the ability to compose skills faster would be very useful for practical LLM finetuning applications.

Weaknesses

- It might be a bit hard to read the trends and compare results in the tables, visualizing some results might be helpful.

Questions

- Line 157 mentions that the data has a form of [prompt 1, answer 1, prompt 2, answer 2]. I do not quite understand why we need prompt 2 and answer 2 here. Will the analysis still hold for simple [prompt 1, answer 1] data samples? - In line 260, the paper mentions a counter-intuitive phenomenon that the held-out performance is better than the train performance. The hypothetical explanation is that the model knows how to compose the held-out skills better than training skills. I think an experiment of switching the skill categories of the training and held-out sets can further support (or against) this hypothesis. - The problem setting studied in this paper reminds me of the “least-to-most” prompting design [1]. This paper shows using some in-context examples of $k=1,2$ can make the model generalize to problems with large $k$. Hence I’m curious about whether combining these more complex in-context prompt designs could further improve the finetuning performance. (I think the analysis and experiment in the current submission is enough for a good paper, just point out this paper and a potential idea.) [1] Zhou, Denny, et al. "Least-to-most prompting enables complex reasoning in large language models." ICLR 2023

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Please refer to the question and weakness part.

Summary

This article studies whether or not LLMs can be fine-tuned to compose skills for text generation. The authors generated training data from GPt-4 in the style of the Skill-Mix benchmark, asking the model to generate text about a topic while using a set of k skills (e.g., sympathy, temporal reasoning, syllogism, etc.). LlaMa-2 and Mistral-7B were then fine-tuned on this synthetic data and evaluated on their ability to generalize to combinations of novel skills, and to more combinations of skills than seen during training. Using GPT-4 as a grader, the authors find that fine-tuning substantially improves compositional generalization.

Strengths

This article has a number of strengths - addresses the important topic of compositional generalization using complex tasks - well-written article - technically sound - evaluation for combining novel skills and combining novel numbers of skills - Two graders were evaluated (GPT-4 and Claude) - Data efficiency was also studied This is solid work that is consistent with other recent work on "learning to compose" through training [1,2]. It addresses these ideas on a larger scale than in past studies and shows compelling results, especially the generalization to more complex combinations. [1] Conklin, H., Wang, B., Smith, K., & Titov, I. (2021). Meta-learning to compositionally generalize. arXiv preprint arXiv:2106.04252. [2] Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985), 115-121.

Weaknesses

The tasks in the Skill-Mix benchmark seem quite artificial, and difficult for people to both produce and evaluate. Only two example generations are provide in the appendix, and the first one has odd syntax in the prompt: "The example should be a short tweet up to a few lines in the context of produce review..." Another weakness is the use of automatic grading by GPT-4, although it seems unavoidable and they also tested another grader from Claude. Additional discussion justifying this choice and the accuracy of the grader seems warranted.

Questions

I don't have additional questions at this point.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

This section is fine.

Summary

The paper explores the capacity of smaller language models to learn compositional generalization from finetuning. Utilizing the Skill-Mix set-up, the study delivers comprehensive experiments to assesse how small language models can improve their performance on both in-distribution and out-of-distribution compositional tasks after finetuning on systhetic datasets regarding in-distribution easier tasks generated by GPT-4.

Strengths

- The paper is well-written. - The paper focuses on a very important topic (compositional generalization) of current LLM research. - It's easy to understand the intuition behind the comprehensive experiments.

Weaknesses

- The experiment pipeline is not novel and highly overlaps with the previous work [1] . - The paper is limited on one specific method on evaluating if language models can generalize compositionally on 'harder' tasks after finetuning on 'easier' compositional examples. There exists many other evaluation methods/metrics covered in related works that are not mentioned, including [2] and [3]. - The paper does not properly explain and support the claim made in line 64-65 and line 243-244, i.e., > Instead, they are acquiring a higher-order meta-skill that allows them to generalize and apply to combine unseen skills. > The results suggest that its ability to compose multiple skills does not come from overfitting training data but should be perceived as learning a *meta-skill* instead. This claim is very strong and needs more experimental results from other compositional tasks or theoretical justifications. [1] Yu, Dingli, et al. "Skill-Mix: A flexible and expandable family of evaluations for AI models." arXiv preprint arXiv:2310.17567 (2023). [2] Dziri, Nouha, et al. "Faith and fate: Limits of transformers on compositionality." Advances in Neural Information Processing Systems 36 (2024). [3] Chen, Jiaao, et al. "Skills-in-context prompting: Unlocking compositionality in large language models." arXiv preprint arXiv:2308.00304 (2023).

Questions

- The skill-mix paper only released small part of topics and skills to prevent people from chasing the leaderboard, so I'm wondering if the full list of skills and topics used in this paper is the same as the original paper. - The second-last line in Table 3 shows that Mistral-7B-Instruct-v0.2 get 0 point for Skill Fraction on $\text { SKILL-MIX }(k)$ after finetuning on $\mathcal{D}_{\text {SKILL-MIX }}(1,2)$. This seems to be a typo. - The experimental results showing performance improvements after finetuning are unsurprised unless the author can justify if the model is indeed learning how to combine skills compositionally during finetuning.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are mentioned in the Appendix section.

Summary

The authors present a (family of) fine-tuned LLMs, trained using the SKILL-MIX dataset. They show that models trained on composite tasks where each instance involves a sequence of different skills improves the model in similar composite tasks. They demonstrate generalization by fine-tuning the models on subsets of task combinations and evaluating them on held-out sets. They also evaluate out-of-distribution generalization on the number of skills sequenced together and show that training on sequences of 2 or 3 skills can help generalize to sequences of 4 and 5 skills.

Strengths

1. The authors have done extensive experiments using the SkillMix dataset to support their arguments. 2. Using LLM to judge LLMs can introduce a lot of variability. I understand that the authors are following the procedure from Yu et al. Reading Appendix B of Yu et al. did increase my confidence and I appreciate adding another evaluator model which supports the analysis even further.

Weaknesses

1. My main concerns with this paper have less to do with the methods of this particulaar submission, but of the dataset and methodology from Yu et al. that the authors extensively draw upon. Looking over the dataset and scoring design, it seems that the scores could be highly inflated simply by writing three disconnected sentences that each individually illustrates the skill. The only criterion in the rubric that is relevant for cohesion is ‘Makes sense’, which still can be evaluated independently for each sentence. This seems to be a rather weak form of skill compositionality that is the central point of this paper. Moreover, the authors imply that the skills and topics would not be highly related (Line 138), but given the skills presented in Appendix B, I’m not entirely convinced that this is true. I would be surprised if there weren’t at least a few hyperlinks between them. 2. The baselines presented in Section 4.1 seem a bit weak. Fine-tuning on SM_train(1) is a useful starting baseline, but it is still severely restricted in its ability to span the dataset. If I understand the authors’ intent correctly that the goal is to demonstrate that fine-tuning to combine different skills, then what I think would be a more informative comparison would be to include within the training set samples involving the same skill/topic category but use higher than 1 k. Otherwise, this confounds with length-generalization, which we know sequence models struggle with. 3. A similar concern exists for Sections 4.2 and 4.3, where training with higher k is confounded with training on longer sequences. I would be interested in understanding the effect size after controlling for sequence length how and what the distribution of sequence lengths are between the different k’s. 4. The three findings are not particularly novel nor surprising (in fact, it would’ve been more surprising if any of the three findings had been shown to be false). The authors appeal to novelty with respect to Arora and Goyal (Line 73), and if that is indeed true, then it would be helpful to provide more substantial analyses with respect to the specific claims made in the first paper (e.g. show which relations or equations from Arora and Goyal fail to predict the results of this paper).

Questions

1. Perhaps I missed this in the main text, but how many seeds and dataset recombinations were used in the experiments? I’m particularly interested in the observation that the model performed better on the held-out set than the training set (Line 260) and whether that is a consistent phenomenon. 2. Could you provide a metric for how consistently the two models from Table 4 rated the responses? It would provide higher confidence if the 31% and 24% (top-left entries) actually had high overlap.

Rating

5

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

The authors have acknowledged the limitations and impact in the supplements.