GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

Summary

The goal of this paper is to utilize locally available models, such as vicuna and opt, to learn the tool utilization capability of black-box models like GPT-3.5. They use the large models to generate instructions and samples using image context and apply instruction tuning on these data with the fine-tunable models. Various data augmentation techniques such as negative samples and context samples were also applied. Furthermore, they propose several automated evaluation methods, including the success rate based on the combination of thought, action, and arguments. Experimental results demonstrate significant improvement in tool invocation effectiveness for their fine-tuned models compared to the original backbone models.

Strengths

1. This paper proposes a method for generating instructions using image context, which enables the large language model to consider visual information when generating multimodal instructions. 2. The method of generating negative data has not been considered in previous works. 3. This paper introduces a novel automated evaluation metric that combines multiple success rates.

Weaknesses

1. The method proposed in this paper, which utilizes self-instruction techniques of large models to enable tool using capabilities for primitive models, is not novel. Similar techniques have been proposed in several works, such as tool-llama (https://arxiv.org/abs/2304.08354), tool-alpaca (http://arxiv.org/abs/2306.05301), and gorilla (http://arxiv.org/abs/2305.15334). The authors have not made significant contributions to the SFT method. 2. One of the main contributions of this paper, image context, is not thoroughly elaborated in this paper. The self-instruction method seems to be also similar to that of MM-react and LLaVA. 3. For the evaluation metrics, the success rate of thought is not clearly explained. Thought should be a natural languge text with certain diversity, but it is unclear whether the generated thought needs to be identical to the ground truth thought to achieve success rate 1. If they need to be identical, I doubt that the success rate of thought would not be as high as shown in the experimental results, as the model may generate thought that is semantically consistent but not textually identical to the groundtruth. If they do not need to be identical, it would be helpful to provide details on how the success rate of thought is calculated. 4. Some recent strong multimodal baselines, such as HuggingGPT (http://arxiv.org/abs/2303.17580) and gorilla (http://arxiv.org/abs/2305.15334), are missing in this paper.

Questions

1. Is there a technical error in Equation 1? where it states that "Y" is sampled from the teacher model based on "X" to obtain the prompt. However, based on the textual description, Equation 1 should refer to sampling the instruction based on "X" and "P_t." 2. How the image context was constructed? 3. How is the success rate of thought be computed?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None

Summary

This work propose GPT4Tools, a self-instruction approach to teach large language models to use tools to solve vision tasks. The authors construct an instruction-following dataset by ChatGPT from the combinations of a set of images and 23 image-related tools. In addition, they augment the dataset to enable to capability to not using tools or using tools in multiple turns. Then they apply Low-Rank Adaptation (LoRA) to open-source LLMs like LLaMa and Vicuna to enable them to use tools. They also propose to evaluate the model's performance on tool-using by measureing the success rate from multiple ascepts. The empirical results show that GPT4Tools can successfully utilize the tools and also have capability to use new tools. The study introduces GPT4Tools, a self-instruction method that teaches large language models (LLMs) to utilize tools for solving vision tasks. The authors establish an instruction-following dataset generated by ChatGPT using combinations of images and 23 image-related tools. The dataset is augmented to allow for the capability of non-usage of tools or multi-turn tool usage. The researchers then apply Low-Rank Adaptation (LoRA) to open-source LLMs, like LLaMa and Vicuna, enabling these models to use tools. The paper also proposes an evaluation measure for the model's success rate from multiple aspects. Empirical results reveal that GPT4Tools can effectively use tools and demonstrate the capability to adopt new tools.

Strengths

- The paper presents an innovative approach to teaching LLMs to use tools for visual tasks through self-instruction - The impact of enabling LLMs to utilize tools could be substantial and pervasive.

Weaknesses

- The paper falls short in providing a comparison with VisProg[1], which also instructs LLMs to use visual tools through few-shot demonstrations. Although GPT4Tools has the benefit of not utilizing proprietary LLMs in inference, a performance comparison would enrich the paper. - The evaluation method appears less compelling because both the training and testing data are generated by using the specific set of tools. Evaluation on standard benchmarks, such as GQA as in [1], could be more convincing. - The paper could benefit from enhanced clarity in its descriptions and explanations. Suggestions include: - Clarifying the variables P_t, X_C, Y, possibly through their corresponding parts in Figure 1 or mentioning corresponding examples in the appendix. - The subscript of P_t could be confusing as the total number of tools is defined as N. - The abundant font colors in Figure 1 and Figure 3 may confuse readers, and it would be helpful to indicate which parts are generated by LLMs - (Minor) Some crucial details appear to be omitted, such as the process of extracting X_C from images, the image source, and the filtering process of the dataset. - (Minor) Typos: - Page 6, line 172: "and and" - Page 6, line 173: ""items.." [1] Gupta, Tanmay, and Aniruddha Kembhavi. "Visual programming: Compositional visual reasoning without training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Questions

- Can GPT4Tools effectively do multi-turn planning, given the multi-turn examples in the augmented data? Why does the evaluation data only contain single-turn examples? - How scalable is GPT4Tools when using a larger number of tools? - (Minor) What the numerical codes (e.g., 179.44, 105.55) represent in the image content shown in Figure 1?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

4 excellent

Limitations

While the authors discuss potential overfitting for longer tuning iterations and the inferior performance of OPT-13B, more detailed discussions regarding the limitations of the proposed evaluation method, the multi-turn capability, and the scalability of GPT4Tools would add value.

Summary

This paper utilizes ChatGPT to generate instruction data for training moderately-sized open-source models to use tools. The authors fine-tune LLMs using LoRA and demonstrate their effectiveness on various visual problems. Additionally, the paper establishes a benchmark for evaluating the LLMs' tool usage ability.

Strengths

1. Using ChatGPT, this paper represents an early effort to consider visual information to produce diverse instruction data that trains moderately-sized open-source models to use tools, resulting in effective learning outcomes. 2. This paper sets a benchmark standard for evaluating the competency of LLMs in tool usage. 3. The results of the experiment demonstrate the efficacy of instruction tuning, as evidenced by the superior performance of the tuned 13B model in comparison to the 175B GPT-3.5 model. Additionally, the study highlights the framework's capacity for generalization to unseen tools.

Weaknesses

1. It would be intriguing to understand the reasons behind the significant discrepancy in generalization ability between OPT and LLaMA in Table 2. 2. The prefix prompt used in the study includes both system messages and tool definitions. Although the current context length of open LLMs appears to be sufficient for the 23 tools examined in the paper, it may not be enough if a broader range of tools is considered. Therefore, the present framework would need to address this limitation if it is to accommodate a more diverse range of tools.

Questions

please refer to Weaknesses

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

please refer to Weaknesses

Summary

The authors propose GPT4Tools, a new approach to data collection for tool use, along with a novel evaluation method. In this approach, ChatGPT is utilized as a teacher to generate instruction-following data for vision-language tasks related to tools. In addition to the single-turn samples, the authors suggest synthesizing negative samples to instruct the student model on when not to use a tool. They also propose using context samples to guide the student model in utilizing relevant tools based on contextual information. Experimental results indicate that by employing the proposed successful rate(s), open source models like LLaMA-13B and Vicuna-13B achieve significant performance improvements after LoRA fine-tuning. Furthermore, the authors demonstrate a certain level of capability in utilizing unseen tools after the fine-tuning process.

Strengths

* The proposed procedure for instruction data collection is straightforward and easy to follow. The performance improvement of the open-source models is evident. The effectiveness of the proposed negative samples and context samples is clearly demonstrated in facilitating downstream tool usage. * The proposed evaluation metric is logical and valuable in establishing a standard for future research in this field. * The conducted ablation study on data augmentation, model size, and tuning iterations provides valuable insights into understanding the behavior of the model in depth.

Weaknesses

* I believe the work focuses on a straightforward procedure for instruction data collection and a standardized evaluation method. Therefore, I don't identify any significant weaknesses, but rather some points for further discussion: * The process of creating synthetic data from ChatGPT and using it to fine-tune smaller models can be seen as a distillation process. Previous studies have demonstrated that, for many tasks, distilled student language models can outperform their teacher language models across various tasks [A1]. What are the authors' thoughts on the relationship between instruction tuning and distillation in the context of tool use? * Vision tasks often exhibit stronger interdependencies. How effectively can the proposed instruction data collection procedure generalize to other domains, such as manipulating a vast number of APIs for cloud services, etc? * I did not find a clear distinction between the two sub-figures in Figure 2. Could the authors emphasize the key area that readers should focus on? * There is a typo on line 155: "single-tune" should be corrected to "single-turn." [A1] Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. ACL 2023.

Questions

Please see the weaknesses section above. Overall I enjoy reading the submission. Since the work mostly discusses instruction data collection procedure, it would be great to expand the discussion between the proposed method and other related fields, such as distillation, other instruction tuning works, etc.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are included

Final review on Submission1848 by Reviewer F8mq

Thank you to the authors for their rebuttal. The authors has explained its difference in using open-source language models to use multi-modal tools and the first work to conduct this. It appears that the authors have addressed the weakness in their response, so I revise my score from 4 to 5. Just as mentioned in Reviewer MACc, I also remain the score at this level duo to that the limited novelty of the proposed method.