Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.
Paper
Similar papers
Peer review
Summary
In this paper, the authors present an LLM finetuned from 65 LLaMa with very limited 1000 samples. LIMA achieves very impressive results in the human evaluation, with competitive results against GPT4/Bard/davinci003. According to my understanding, the main contribution of this paper lies on two aspects: 1. verifying fact that carefully constructed samples enhance LLM more; 2. deriving a well-constructed SFT dataset as well as a strong LLM.
Strengths
1. The paper verifies a fact that carefully constructed samples enhances LLM more 2. The presented model is a very strong competitor to existing non open-source LLM 3. The extensive analysis on data diversity, quality and quantity is useful.
Weaknesses
1. Quality experiments are limited, where its empirical results are mostly from human evaluation on self-constructed dataset; 2. The data construction process is hard to scale 3. The evaluation is conducted on very limited data (~300)
Questions
The claim of this paper is very strong and impressive. I think such a strong claim should be supported by aa more thorough experiments. 1. The test set is from pushshift r/AskReddut and authors' questions. I would be helpful to see its performance under a finegrained evaluation on its capacability, such as knowledge QA (literature, science), reasoning, toxicity, coding and etc. 2. Human evaluation is subjective. I wonder if the authors can provide results with objective metrics. 3. Out-of-istriution analysis is interesting. Is it possible to evaluate LIMA on questions from a third party (e.g. questions from sharegpt)? 4. LIMA is tuned with 1000 samples. Do you try other number of sampels? I wonder how LLM performance changes against increasing quantity of well curated samples. How do you determine the number of well curated samples set for a certain model? And is the conclusion consistent across base models? If my concerns are addressed, I am happy to raise my score.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
4 excellent
Contribution
3 good
Limitations
Yes
Summary
This paper presents a superficial alignment hypothesis. It claims that most knowledge and abilities of large language models (LLMs) are learned in the pre-training stage, and a few datasets are just enough to transform the styles of LLMs for alignment. To support the argument, the authors of the paper finetune LLaMA on the few curated finetuning dataset only having 1000 instances. The curated dataset consists of community-based Q&A such as Stack Exchange, WikiHow, and Reddit, and manually authored examples. Experimental results (human evaluation on 300 test samples) show that the resulting model, LIMA outperforms baselines including Alpaca 65B, Davinci003, and Bard, and shows compatible performances with other proprietary LLMs, Claude and GPT-4. Further analyses confirm the results qualitatively, showing generalizability on the safety or multi-turn scenarios. Moreover, the data quality and diversity are suggested to be the most crucial factor for the alignment.
Strengths
1. The hypothesis is convincing, and the experimental results support it well. 2. The main message is clearly delivered. 3. Evaluations and corresponding cross-agreements are well conducted.
Weaknesses
Overall the evaluation is limited. - a. More details for the manually designed test prompts are needed, e.g., category, distribution, etc. - b. Human (GPT-4) judgment does not guarantee the intended behaviors of the models, according to [1]. - b-1. There are automatic evaluation benchmarks related to alignment learning, such as HHH evaluation in Big-Bench, TruthfulQA, and RealToxicityPrompt. - b-2. Also conventional NLP tasks can also be considered as the target of measurements. Specifically, knowledge probing (MMLU or Reasoning tasks) is a critical factor in getting the model, not just mimicking the preferred styles. - The number of examples for the safety or multi-turn analysis is too small. Therefore, I think the claim for the multi-turn dialogue is not convincing, and the analysis should be based on at least 50-100 examples. - Ablation study is conducted on the 7B model, unlike the main experiments based on the 65B model. We could not guarantee the same tendency would be observed with the different-sized backbone. [1] The False Promise of Imitating Proprietary LLMs (Gudibande et al.)
Questions
1. The superficial alignment hypothesis is still valid in the small-sized models? Although you reported the ablation study based on the LLaMA-7B, we do not know that the model generates better responses compared to other baselines.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
- It would be good to include evaluation results on other automatic evaluation benchmarks (I suggested in the Weakness). - More details for the manually designed prompts (for the test) make the paper more concrete.
Summary
The authors investigate on instruction tuning an LLM on high-quality manually collected/created data (only 1K training examples). They provide detailed explanation on how to collect the data, tune a 65B LLaMA model and compare it with other commercial/open-source LLMs. The main experiment results show that the LIMA 65B LLaMA model is better than 65B Alpaca and slightly worse than the top commercial models (GPT, Claude and Bard), based on human and GPT-4 judgements. The authors also conduct other ablation studies including dialogue modeling ability, safety, data quality v.s. quantity.
Strengths
1. The presented data quality is very high. There are many recent papers about instruction tuning on a large amount of instruction data (typically by prompting GPTs), while this paper demonstrates that quality can play a bigger role. This finding is very inspiring for the community. 2. The LIMA model can also somehow reject unsafe queries, which is difficult for Alpaca, Vicuna or other fine-tuned LLaMAs on GPT generated data. This result is surprising to the reviewer as the ability of rejection to unsafe query is more likely to gain from RLHF. 3. The model also show promising performance on dialogue tasks. This is good but actually expected, as Vicuna showed that dialogue ability is not hard to gain.
Weaknesses
1. The test set is kind of small. In addition, although the authors mention they put effort to make the test data diverse and distinct from the training set, the diversity of the test set may still be a issue since all the authors are likely to fall within a narrow scope (i.e., well-trained machine learning researchers). It would be better to include more testing prompts from different people, e.g., real user inputs. This might be much harder for academia folks though comparing with OpenAI and Google. 2. As pointed out in [Gudibande et al.](https://arxiv.org/pdf/2305.15717.pdf), the SFT tuned LLaMA models can receive good ratings comparable to ChatGPT from crowd workers but actually have a huge gap, and they called it as a false promise. The evaluation of the LIMA model is very limited to human/GPT judgments. Whether it has the similar issue is unknown from the current presentation. Maybe it's better than LLaMAs tuned on GPT generated data; or it also has the same issue --- which may indicate that RLHF's importance. Current evaluation cannot provide the readers enough information for this. 3. Many recent papers have shown that the real emergent ability, aka what makes GPTs/Bard/Claude different from the others are complex task solving. See [Yao et al.](https://yaofu.notion.site/Towards-Complex-Reasoning-the-Polaris-of-Large-Language-Models-c2b4a51355b44764975f88e6a42d4e75), and also OpenAI’s [GPT-4 release blog](https://openai.com/research/gpt-4): “In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when the complexity of the task reaches a sufficient threshold”. The reviewer would like to see the analysis on these kind of tasks (e.g., MATH/BB-Hard) as well.
Questions
In line 78, "Within each exchange, we take the questions with the highest score that are self-contained in the title (no body)". The reviewer doesn't understand this sentence. Could you clarify the meaning of the terms here? What is the score for, what is "no body" and what does self-contained mean here?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The way of manually collecting high quality data could be expensive and hard to scale. It's still a question that given the same amount of money, which option is more effective: manually collect data and guarantee quality; or just prompt powerful closed-source LLMs to get a huge amount of data.
Summary
This paper presents LIMA, a 65 billion-parameter large language model fine-tuned on a dataset of 1,000 carefully curated prompts and responses without the use of reinforcement learning or human preference modeling. The aim of this research was to measure the relative importance of pretraining and large-scale instruction tuning in the training of language models. LIMA demonstrates strong performance and generalizability across a variety of tasks, even those not included in its training data. It also excels in specific response formats, learning from just a few examples in the training data. The study includes a controlled human evaluation where LIMA's responses were either equivalent or preferred to those of GPT-4 in 43% of cases. When compared to Bard and DaVinci003, this percentage rose to 58% and 65%, respectively. The results suggest that the majority of knowledge in large language models is acquired during the pretraining stage, and only limited instruction tuning data is required to generate high-quality output. Ablation experiments underscored the diminishing returns from scaling data quantity without a concurrent increase in prompt diversity, highlighting the critical role of data quality and diversity in training these models.
Strengths
1. The paper presents an effective approach to training large language models, focusing on the importance of pretraining and using a limited number of carefully curated prompts for fine-tuning. This approach challenges the common practice of using large-scale instruction tuning and reinforcement learning. 2. The experimental results indicate the effectiveness of the proposed training approach, which shows strong performance compared to various LLMs, including commercial ones. 3. Detailed ablation studies are given regarding the data diversity, quality, and quantity, which allows the readers to better understand different aspects of the proposed approach (and data).
Weaknesses
1. The proposed training scheme is only tested with subjective evaluations. I wonder if the proposed method can also give significant improvements on objective benchmarks, such as MMLU, etc. 2. Also, the evaluation can be further improved by introducing different settings, such as CoT, few-shot, etc. 3. It is not quite clear how the conclusions can be applied to other scenarios, such as different languages, domains, etc. The paper can be further strengthened by providing some takeaways to the readers.
Questions
Question: 1. Please address the weaknesses above. 2. Will the 1,000 examples and LIMA model be released to the public to further promote open research? 3. As the training process only involves 1K examples, why the experiments are only performed on 65B LLaMA? I know that the ablations are carried out on 7B model, but it would be also interesting to see a progressive analysis when model scales from 7B, 13B, 33B to 65B. Other comments: There are some typos and grammatical issues that should be corrected. 1. line 2: large scale -> large-scale 2. line 37: LLaMa -> LLaMA 3. line 163: state of the art -> state-of-the-art 4. line 206: Of -> of
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors discussed their limitations in Section 7.
Summary
This paper hypothesizes that alignment of pretrained LLMs to become user friendly and useful AI chatbots is mainly about learning the style or format of interacting with users, and that all/most knowledge and capabilities were already acquired by the LLM during pretraining. To test this hypothesis, this paper curates 1,000 diverse high quality examples that approximate real user prompts along with high-quality responses, and trains a new model called LIMA. LIMA demonstrates good performance in a controlled human study; responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback.
Strengths
This paper addresses an important and timely question aiming to shed light on the mechanisms behind the huge success of instruction tuned LLMs. The paper is very well written and easy to follow. I really like the straightforward approach of the paper: it clearly defines the crisp question regarding the role of instruction tuning, and opts to answer it directly by creating a small, clean, and diverse training set. The resulting model is clearly a very good instruct model, and achieving this so simply is a very impressive feat. Judging this paper from an academic standpoint, I do have some concerns, detailed below.
Weaknesses
If I understand correctly, table 1 implies that over 75% of the test set is written by the paper authors (group B) which received the same general instructions as the paper authors who wrote 20% of the training set and the entire dev set questions (group A). I see this as weakening the paper’s messages, in particular the comparisons to other models in figures 1 and 2 could be unclean. One way to put it, as I think the paper is trying to do, is that group B are different people than group A with different interests, friends, etc, and therefore the test questions of group B can be considered an unrelated ground truth test. Another way to put it, which I think is valid, is that LIMA enjoys a significant amount of training on data that is qualitatively closer to the test distribution than any of the data used for training any of the other models. This is concerning since LIMA can be said to enjoy significant training on the tets distribution while other models don’t; this is known to have dramatic effects. An ablation that slices the results of figure 1 according to 230 test examples written by group B and 70 test examples from Pushshift r/AskReddit can ease my concern here; if the two groups exhibit similar trends, then training on data from group A did not bias the results. If performance on the 230 test examples written by group B is way better, then I think the messages need to be revised. Another experiment that could ease my concern is training “clean-LIMA” without the data from group A, and supplementing it with 200 examples from the other sources of the training set. A third alternative, perhaps better than the former, is to devise a "clean test set" without examples written by group B. I realize that the last two require further annotation and may be harder, but the authors did show that automatic evaluation with GPT-4 correlates with human judgement and this can be a place to leverage that. If the authors convince me that I either misunderstood their setup or that training from group A did not bias the test results,I’d be happy to raise my soundness score and overall evaluation.
Questions
How do you fine-tune Llama on the Alpaca dataset? Is there a standard setting which the authors released? Did you perform a hyper-param search? Since this is the only baseline model that LIMA beats, and the only one that you trained in-house, it is important to make sure that it receives at least equal attention in training as your model. I’d be happy if you could elaborate exactly on how you trained both LIMA and your version of Alpaca.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Yes
Replies and Further Questions
> Weakness 1: The test set does indeed include naturally occurring prompts from Reddit. Given the portion of examples from reddit (~20+%) and from authors (70 from reddit v.s. 230 from authors), the concerns in the original review still stand. Reporting a separate number for the reddit subset can be the simplest way to address this. > Weakness 2: We agree that there are limitations to evaluating models by human preferences. However, there are even more significant limitations to evaluating models by other existing benchmarks, e.g. MMLU, which are completely misaligned with the actual usage of assistants/chatbots. Please provide a constructive suggestion for evaluating this type of functionality. There are many ways to do so in both academia and industry. For example, the HHH principle (Helpfulness, Faithfulness, Harmless). Different aspects of the result quality, e.g., task following, creativity, fluency etc. A simple "model x beating y" means little in the actual use case and is far from enough. There are many papers discussed these topics and benchmarks available, e.g., see Anthropic's paper series. > Weakness 3: Indeed, but these are all examples of advances in pretraining, which is not in the scope of this paper. LIMA focuses on showing that the stages following pretraining can be greatly simplified. The emergent ability phenomenon also exists in instruction tuning. See Sec 4 in [Wei et al.](https://arxiv.org/pdf/2206.07682.pdf). I may provide wrong example tasks though. > The way of manually collecting high quality data could be expensive and hard to scale. It's still a question that given the same amount of money, which option is more effective: manually collect data and guarantee quality; or just prompt powerful closed-source LLMs to get a huge amount of data. No comments addressing this yet. Accessing human labels are expensive, especially when requiring high quality annotations. Given the same amount of money, it's actually possible that calling APIs can generate an instruction tuning dataset with lower quality but still achieves similar outcome.
No problem! We actually wrote this for reviewer a3cj, but here is the result table copied over: | | Pushshift r/AskReddit (70) | Authors (230) | ALL (300) | |-------------------|-----------------------------|---------------|----------------| | LIMA vs Alpaca (65B) | 64%/16%/20% | 50%/22%/28% | 53%/21%/26% | | LIMA vs DaVinci003 | 57%/13%/30% | 40%/23%/36% | 44%/21%/35% | | LIMA vs BARD (April) | 30%/29%/41% | 33%/24%/43% | 33%/25%/42% | | LIMA vs Claude (April) | 20%/27%/53% | 25%/20%/55% | 24%/22%/54% | | LIMA vs GPT4 | 22%/17%/60% | 17%/27%/56% | 18%/25%/57% | As we can see from the table, LIMA’s performance on AskReddit prompts is indeed similar to its performance on Group B’s prompts. In fact, when compared to Alpaca or DaVinci003, performance on AskReddit is significantly better. Alternative Evaluations: The HHH principle promoted by Anthropic is interesting, and certainly has merits, but it is only way of defining "aspects" or "dimensions" by which to compare models. It is not necessarily the ground truth. Scaling Data: Actually, distilling data from APIs is scientifically problematic - it assumes that a "gold" model already exists. We are interested in finding an efficient recipe for creating aligned language models without dependency on other aligned models.
Raising my score
I thank the authors for clarifying the issue. Indeed, it looks like the similarity between groups A and B did not bias the results. I am raising my score accordingly.
Thanks for the updates. I will keep my score.
Thank you for your responses, especially for the prompts category. I raise my score accordingly.
Decision
Accept (poster)