Instruction Tuning With Loss Over Instructions

Summary

This paper proposes a new method called Instruction Modelling (IM) for training language models, which applies a loss function to both the instruction and output parts of training data, rather than just the output. Through experiments on diverse benchmarks, the authors show that IM can improve model performance on both NLP tasks and open-ended generation benchmarks compared to standard Instruction Tuning (IT). The effectiveness of IM is influenced by two key factors: the ratio between instruction length and output length in the training data, and the number of training examples. IM is particularly beneficial for datasets with long instructions paired with brief outputs, or when using a small amount of training data. The authors hypothesize that IM's improvements stem from reducing overfitting during instruction tuning.

Strengths

- The paper revisits the fundamental approach to instruction tuning, which typically involves calculating loss only on the output portion of the data. By proposing Instruction Modelling (IM), which applies loss to both the instruction and output parts, the authors challenge this standard practice. This fresh perspective on a widely-used technique is a key strength of the paper. - The authors conduct extensive experiments across many diverse benchmarks, demonstrating the broad applicability and effectiveness of their proposed method. - The paper identifies and analyzes two crucial factors influencing IM's effectiveness: the ratio between instruction length and output length in training data, and the number of training examples. This analysis provides valuable insights for practitioners on when and how to best apply the IM approach, particularly in low-resource scenarios.

Weaknesses

- Similar ideas and conclusions have been proposed by previous work [1]. - While the paper presents empirical results showing the effectiveness of Instruction Modelling (IM), it lacks a strong foundation explaining why applying loss to instructions works. The authors hypothesize that IM reduces overfitting, but a more rigorous theoretical analysis could provide deeper insights into the mechanism behind IM's success. - The experiments primarily use LLaMA-2 and OPT models. While these are significant models, the paper doesn't explore how IM performs across a wider range of model architectures or sizes, e.g., whether the conclusion also holds for a 34B or a 70B model. [1] Instruction Fine-Tuning: Does Prompt Loss Matter? https://arxiv.org/pdf/2401.13586v2

Questions

- Are there any possible experiments that you can do to further explain why applying loss to instruction works? - Do you think when we have a significantly larger size of instruction tuning data, the conclusion still holds? For example, we see the recent release of Llama-3 models, which adopted 10+ million instructions. And which factor is more important: the ratio between output and the instruction length or the number of instruction tuning samples?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. But the authors do not have a mandatory paper checklist.

Summary

The propose that when updating models using instruction tuning, the models should also be updated based on loss on the instruction itself. This is a simple change that is un-intuitive, so the successful results are impressive.

Strengths

They include experiments from many different datasets and look at multiple different LLMs. And there results generally hold across them.

Weaknesses

It is unclear what the paper considers an "instruction", there should be some examples of it. At one point it says that "static parts" of the instruction, like a "<user>:" token, are masked from the lost. Does that mean the rest of the instruction is all examples? An instruction like "Tell me the sentiment of this text" is going to be static across examples. When looking at table 1, although their IM methods wins in terms of average performance, there are many datasets where it performs worse. This is glossed over in their prose. Their "Loss analysis" experiments are not convincing as they don't fully capture the changes to the model that occur in each setting. By only checking the loss on the continuations, they don't capture how their IM model might be more overfit to the instructions themselves. For example, if you looked that the train loss for the whole Instruction + continuation for each models, the IM model would mostly likely be far lower than the loss for the instruction tuned model. It makes sense that the IM model is going to be worse on the continuations after the same amount of training as updates to get better as the instruction itself will cause conflicts. Discarding the models performance on the Instructions creates a self-fullfilling prophecy of the loss being higher when only the continuations are considered.

Questions

What is considered an "instruction" when training? The text mentions that "static" tokens like "<user>:" are masked out, but many instructions are static, i.e. "Tell me if the second sentence entails the first". If instructions are non-static are they mostly example like in In-Context-Learning? They mention that IM is most effective in settings where the instructions are long. Is it possible that most of this gain is for essentially continuing pre-training on in-domain data a la https://arxiv.org/abs/2004.10964 Similarly, do you think the success in the SAH settings is possibly an artifact of getting a model that is just trained on a lot more data? How many more tokens does an IM model see than in IT model in that setting? Is there a way to hold the number of tokens seen constant? For example by training the IM model on even fewer examples to show that the gain really is from the training on the instruction?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

yes

Summary

This paper proposes Instruction Modeling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. The method is found to be effective on NLP tasks and open-ended generation benchmarks. This paper found two key factors that influence the effectiveness of the approach: (1) the ratio between instruction and output lengths, and (2) the quantity of training data. There are also additional analysis that shows IM can reduce overfitting.

Strengths

Generally the paper is well written and covers a wide range of experiments. It forms a comprehensive study on whether insturction tuning should calculate loss over the instruction part. Actually, it's a little surprise to me that calculating loss over the instruction part can improve the performance of the model, and the reasoning behind this explained by the paper is insightful and interesting. Overall I think: 1. The finding is interesting and novel. This is in contrast to the common practice of calculating loss over the output part only. I think it's going to have broad impact on how people finetune LMs in the future if all the claims are true. 2. The experiments and evaluations are comprehensive. THe paper covers a wide range of instruction datasets and quite a few widely used benchmarks. 3. The paper has a lot of details, in both the main body and the appendix. The appendix is very detailed and informative, which is good for reproducibility.

Weaknesses

A part I feel that's missing in the paper is analysis on the **quality of the instruction part**. One reason that people didn't calculate loss over the instruction part is that the instruction part is usually noisy and not well-structured, e.g, ShareGPT has a lot of user shared low-quality instructions. The paper should analyze how the quality of the instruction part affects the performance of the model. Intuitively, if the model is also learning from the noisy instruction part, it's possible that the model will learn undesirable patterns from the instruction part, but I don't see this being discussed in the paper. Moreover, the number of LMs being finetuned and tested is relatively small. It will be better if more families of LMs and larger models (eg, 60-70B params) are tested.

Questions

1. Do you have any analysis on the quality of the instruction part? How does the quality of the instruction part affect the performance of the model? See "Weaknesses" part. 2. I might have missed this one in the paper, but how do you pack multiple samples during finetuning? Do you concatenate samples to form a fixed length and then trucate them? Or do you add padding tokens? 3. Do you have any scenarios where the IM is significantly worse than the baseline? If so, can you provide some examples? I want to know under what conditions IM is not effective and should be avoided. 4. Do you think IM can be used as a drop-in replacement for the current finetuning process? Or do you think it should be used in conjunction with the current finetuning process?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are being discussed in the paper.

Summary

In this work, authors propose to use instruction modeling (using loss over the full instruction-output pair) instead of just instruction tuning (using loss over the output given the instruction) as a method for supervised finetuning on LLMs. The authors demonstrate consistent gains over multiple benchmarks using this simple technique and hypothesize that the gains occur due to reduced overfitting to instruction tuning dataset.

Strengths

1. The proposed method is quite simple and scalable as such would be of great interest to practitioners in the community. 2. The authors try to characterize the gains and attribute it to the extra signal obtained in the case of short outputs as well as to reduced overfitting of the approach. Both characterizations seem intuitive in explaining the results obtained. Further, the authors show that proposed method is complementary to NEFTUNE, another recent method which results in consistent empirical gains.

Weaknesses

1. Limited Models Explored: While the authors do a good job of experimenting with multiple instruction datasets, the experiments are only done with LLama-2 (7B and 13B) and OPT-7B models. This forgoes a plethora of open source models, experiments on which would have greatly strengthen the paper. Given that these LLMs are all from the Meta family of LLMs -- this raises some suspicion as to the generality of the proposed technique when different architectures or modeling assumptions are made.

Questions

1. Do you have any results on other LLM families: Phi, MPT or Gemma?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitations are adequately addressed.