SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection

Summary

The paper introduces SelectIT, a novel instruction tuning (IT) method that leverages the intrinsic uncertainty in large language models (LLMs) to select high-quality IT data without requiring additional resources. The authors present a curated IT dataset, Selective Alpaca, derived from the Alpaca-GPT4 dataset using SelectIT. The empirical results demonstrate significant improvements in model performance across various tasks. The paper highlights the robustness and efficiency of SelectIT in enhancing the capabilities of LLMs.

Strengths

- The approach of using the intrinsic uncertainty of LLMs for IT data selection is innovative and eliminates the need for external resources, making it cost-effective and widely applicable. - The submission is technically sound, with well-supported claims through theoretical analysis and comprehensive experimental results.

Weaknesses

- Figure 2 is not clear to me. I suggest giving a brief description of each selection stage in the caption for easy understanding. - It seems like the foundational LLM's capacity becomes the key for this data selection. In this case, the selection method may not be as effective as other external resource methods.

Questions

- How does SelectIT perform when applied to instruction tuning datasets other than Alpaca-GPT4? - Can the authors provide more detailed examples or case studies illustrating the differences between data selected by SelectIT and other methods? And how these different influence the final model performance? - How does the performance of SelectIT change with different values of the uncertainty factor (α) ?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

In this paper, authors leverage LLMs to score the training data in terms of the token and sentence and model. They train various models on their select models to show the effectiveness on various benchmarks. Their main contribution is to apply foundations to judge the quality of training data from different granularity without any external resources.

Strengths

1. Their idea is easy to understand and the writing is pretty clear to me. 2. They conduct a lot of experiments on several benchmarks to show the effectiveness.

Weaknesses

1. From the table 3, it seems sentence-R works the best so what if we trained model on the data just from the sentence-R? BTW, what is the ID 6, 7, 8 means in that Table? 2. My main question is that do we really need all three types selection models, can one or two of them already achieve great performance? 3. What is the overhead of this data selection model? Like how long does it take to finish the process on 10K dataset.

Questions

Please refer to the above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper presents SelectIT, a new way to improve how large language models (LLMs) follow instructions using their own uncertainty to pick better training data. This method is cost-effective as it doesn't require extra tools or data. The authors developed a new dataset called Selective Alpaca using SelectIT, which significantly improved LLMs' performance. The study suggests that using longer and more detailed data could be more beneficial for training models.

Strengths

SelectIT is a bew and unique method that improves how large language models (LLMs) learn by using the model's own uncertainty to choose better training data. This approach is special because it doesn't need extra tools or data, making it less expensive and easier to use. The paper includes detailed studies that show the method works well and supports its findings with strong evidence. SelectIT significantly improves how models perform compared to older methods, showing that it's a big improvement in teaching models. The researchers made their methods and data public, allowing others to use and test them. This openness helps the scientific community to work together and trus the results more. The paper is also written in a way that's easy to understand, which helps more people learn about SelectIT. It discusses how using longer and more detailed data can make models learn better. Additionally, the paper compares the costs of using SelectIT with using other common tools like the GPT-4 and ChatGPT APIs, showing that SelectIT can save money. This shows that SelectIT is not only effective but also cost-efficient for training models.

Weaknesses

The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources.

Questions

"The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources. " Clarification? Could you consider applying the SelectIT method to larger datasets, such as the Orca dataset for ChatGPT with 5 million samples and the GPT-4 dataset with 1 million samples? This would help to evaluate whether your method scales effectively with increased data volume.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

the paper lacks a separate section on limitation of the approach.

Summary

The paper introduces a data selection approach to instruction tuning (IT) for large language models (LLMs) by leveraging the intrinsic uncertainties within the models themselves. This method, termed SelectIT, aims to enhance IT data selection without the need for additional external resources, making it more cost-effective and accessible. SelectIT utilizes three levels of uncertainty within LLMs—token-level, sentence-level, and model-level—to effectively rate and select high-quality IT data. The paper demonstrates the effectiveness of SelectIT through empirical results, showing improvements in model abilities on the open-instruct benchmark and other domain-specific tasks. Additionally, SelectIT is highlighted as a faster and more resource-efficient method compared to traditional approaches, making it a more practical solution for widespread adoption. --------------------------------------------------------------------------------------------------------------------------------------------------------------- Thank you for your reply and I have updated my score accordingly.

Strengths

**Originality**: The paper presents a data selection approach to instruction tuning by leveraging the intrinsic uncertainties within LLMs, without relying on additional fine-tuned models or API calls of ChatGPT or Claude. **Quality**: The methodology has a clear explanation of the multi-granularity self-reflection technique. The empirical results are robust, demonstrating the effectiveness of SelectIT across various benchmarks and tasks. **Clarity**: The paper is clearly written, with well-organized sections and comprehensive explanations of the proposed method and its implementation. **Significance**: SelectIT addresses a critical limitation in the field of instruction tuning by removing the dependence on additional models or data, making the approach more cost-effective and scalable. The curated Selective Alpaca dataset offers performance improvements for LLMs, highlighting the practical impact of the research.

Weaknesses

1. **Reliance on Tuning K**: The method depends on tuning the parameter \( K \), which can significantly affect model performance. This dependency introduces additional complexity and may require extensive experimentation to optimize. 2. **Use of Multiple Base Models**: Although the paper claims not to use external resources, it actually relies on multiple different base models to obtain the model-level score. This reliance contradicts the claim of being purely self-reflective and can be confusing, as the title suggests the method is entirely self-contained. 3. **Sentence-Level Quality Evaluation**: The sentence-level quality evaluation requires multiple rating prompts. It is unclear whether other methods, such as AlpaGasus, would achieve similar performance if they also used multiple rating prompts. This comparison is missing and could be critical for understanding the true advantage of the proposed method. 4. **Single Dataset Evaluation**: All evaluations are conducted on a single training dataset, making it questionable whether this method can be generalized to other datasets, such as WizardLM, or multi-turn data, like the ShareGPT dataset. Testing on a more diverse set of datasets would strengthen the paper's claims of robustness and general applicability.

Questions

(1) How the α = 0.2 in the experiments is selected? (2) Will optimal K be training dataset dependent?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

See more details in Weaknesses section.