Summary
The paper presents a framework for enhancing the quality of instruction datasets used for tuning large language models (LLMs). The proposed framework, Star-Agents, leverages multi-agent collaboration to generate, evaluate, and refine instruction data automatically. The approach comprises three main components:
1. **Diverse Data Generation**: The framework employs multiple LLM agents to generate varied instruction data. Each agent pair, consisting of distinct instruction and response agents, creates diverse data samples to enrich the dataset.
2. **Dual-model Evaluation**: This component introduces a two-tiered evaluation mechanism using both small and large models. The evaluation metric considers the difficulty and quality of the generated data, ensuring it is challenging yet manageable for the target model.
3. **Dynamic Refinement**: The framework dynamically adjusts the sampling probabilities of agent pairs based on their performance, promoting the selection of the most effective agents for data generation.
Empirical studies demonstrate the optimized datasets generated by Star-Agents led to performance improvements, with an average increase of 12% and up to 40% in specific benchmarks such as MT-bench, Vicuna-bench, and the WizardLM testset.
------------------------------------------------------------------------------------------------------------------------------------------------------------
Thank you for your reply and I have updated my score accordingly.
Strengths
**Originality**
The paper introduces a framework, Star-Agents, for automatic data optimization in instruction tuning for large language models (LLMs). This originality arises from:
- **Multi-Agent Collaboration**: Utilizing multiple LLM agents to generate diverse and high-quality instruction data. This approach addresses the common limitations of single-model data generation methods, ensuring a richer and more varied dataset.
- **Dual-Model Evaluation**: Implementing a dual-model evaluation strategy that assesses both the difficulty and quality of the generated data. This innovative metric ensures the data is challenging yet manageable, enhancing the instruction tuning process.
- **Dynamic Refinement**: The dynamic adjustment of sampling probabilities for agent pairs based on performance is a creative mechanism that optimizes the data generation process over time.
**Quality**
The paper is supported by empirical studies. The framework is tested using instruction-tuning experiments with two models, including Pythia and LLaMA. The results consistently show performance improvements, validating the quality of the proposed method.
**Clarity**
The paper is clearly written and well-structured, facilitating easy understanding of the proposed framework.
Weaknesses
**Limited Dataset Evaluation**
The paper evaluates the Star-Agents framework on a limited set of datasets, which may not fully capture the framework's robustness and generalizability. Specifically:
- **Small Evaluation Datasets**: The datasets used for evaluations are relatively small. Evaluating the framework on larger, more diverse datasets would provide a better understanding of its effectiveness across different data scales and domains.
**Complexity of the Method**
The Star-Agents framework involves multiple stages and the use of several teacher models, which introduces complexity:
- **Multiple Teacher Models**: The requirement for multiple LLM agents increases the computational cost and complexity of the implementation. Simplifying the framework or exploring methods to reduce the number of required teacher models could enhance its practical applicability.
- **Three-Stage Process**: The three-pronged approach, while thorough, may be overly complex for some applications. Streamlining the process without sacrificing performance gains could make the framework more accessible and easier to implement.
**Lack of Human Evaluations**
The paper does not include human evaluations of test data, which is crucial for assessing the practical quality and usability of the model.
"Absence of Standard Benchmark Results"
The framework's performance is not evaluated on widely recognized benchmarks, limiting the comparability of its results:
- **Standard Benchmarks**: Including evaluations on standard benchmarks such as GSM8K, HumanEval, or other well-known datasets would provide a clearer comparison with existing methods. This would help situate the Star-Agents framework within the broader context of instruction tuning and LLM performance.
Questions
What will be the results if base models are larger, e.g. llama 13b or llama 70b?
Limitations
See more details in Weaknesses section.