Gorilla: Large Language Model Connected with Massive APIs

Summary

This paper proposes Gorilla, a fine-tuned LLaMA model, outperforms GPT-4 in crafting API calls and adapts well to document changes with a document retriever, reducing hallucination issues. It also provides APIBench, a new dataset for evaluation, includes APIs from HuggingFace, TorchHub, and TensorHub. Gorilla's integration with a retrieval system enhances tool usage accuracy and documentation updates, promising more reliable LLM outputs.

Strengths

1. The paper trains a system which connects massive APIs and takes text instruction to get the corresponding API calls, along with a step-by-step explanation of the pipeline. It focuses on the details within an API call, which significantly mitigates the complexity involved in developing machine learning softwares. 2. The paper also constructs APIBench by scraping a large corpus of ML APIs and developing an evaluation framework to check functional correctness. It uses AST subtree-matching metrics which helps measure hallucination and further contributes to the evaluation of ML API mastering techniques. 3. The paper is well-writtened, presenting a detailed comparison with other related works, a clear structure on methods, and comprehensive experiments and analysis.

Weaknesses

1. Although the paper overall does not present significant issues, I would appreciate seeing a performance comparison with more specific methods, such as other works that also utilize LLM for tool calling to build software pipelines, which may further demonstrate the contribution of techniques presented by Gorilla. 2. The paper needs to better illustrate the contributions of this work and more clearly outline its potential impact on the community. 3. Some writing detales need to be improved in the paper, such as "the" should be omitted in "The model trained with the our method" in Line 31, and "AST" should be replaced by "Abstract Syntax Tree" for the first time it appears in Line 36.

Questions

1. In Figure 2, why GPT-3.5 is much better than GPT-4 in all settings? 2. As most of the instructions are generated by GPT-4 referring to your hand-generated instruction seeds. How to ensure your seeds meets the real-world scenario and reach the real difficulties? 3. As many APIs have overlapping functions, how to decide which API is better than the other similar one? How to maintain stable API calling to persist consistent results for the same request? 4. Which Llama version is used as your foundation model? Llama2 or Llama3? 5. In Table 1, why Gorilla always produces a large error in selecting wrong APIs? Do the results support your view?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper should provide a more comprehensive limitation part to point out potential overlooked issues, such as influence of synthetic training data and insufficient comparative experiments.

Summary

The authors present a new fine-tuned language model that is trained to map from user instructions to code snippets that invoke the appropriate APIs. The authors also introduce a new dataset which includes the information for roughly 1600 APIs from a variety of online sources, which is used to train the model. They conclude that their approach outperforms existing language models on the task of API call generation, as measured by both error rate and a novel “hallucination” metric.

Strengths

I feel that the problem statement is well-motivated and of general interest. Prior work has investigated allowing LLMs to make use of various tools, and improvements in that capability are likely to be well received. I appreciate that the authors have chosen a (relatively speaking) lightweight and open-source model, which increases the usability of their approach. I also appreciate the construction of a novel dataset, which could presumably be of interest to the larger community even in the absence of a specific model.

Weaknesses

First, I wonder about baselines. The authors have done a good job of comparing their method against a variety of open-source and closed-source LLMs, but these systems are all “generalist” compared to the fine-tuned Gorilla model. It would have been useful to see a comparison to a fine-tuned version of another model (even one with less fine-tuning than is possible on the small Llama-7b) just to have a sense of how much improvement can be leveraged from the newly introduced dataset. I also wonder if it would have been possible to compare to specialized tool-based LLM models like Toolformer. If such a comparison is not appropriate or possible, I feel the authors should mention why. My second question concerns statistical significance. The authors indicate the high cost of LLM experiments as the reason for omitting such analysis. I sympathize with this explanation, but feel that I would be remiss not to stress the importance of statistical testing as part of justifying claims that Gorilla’s “performance surpasses prompting the state-of-the-art LLMs in three massive datasets.” Without a measure of a variance, we cannot meaningfully conclude that apparent improvements achieved by Gorilla (which are often on the order of 1-3% in overall score) are the result of anything other than noise. To be sure, a 7B parameter model that achieves even comparable results to a massive LLM is a notable result (though see my earlier point about fine-tuned baselines), but I would temper claims about improvement in the absence of statistical justification. Lastly, I feel that the paper would benefit greatly from a more robust error analysis. In the best-performing realistic setting (i.e. without access to an oracle), Gorilla achieves an overall accuracy of roughly 65%. This means that more than one third of the time it is returning an incorrect result (including errors, the authors point out, that could be silent to the end user). Obviously this is far from being ready for direct deployment without external validation. Given the prevalence of errors, it would be useful to better understand what kinds of errors are common: are they more likely to occur with particular kinds of APIs? Does the wording and / or complexity of the input prompt have a large effect? This is the kind of analysis that, in combination with the novel dataset, could help spur future work and improvement. In addition, given the potential for damaging errors, a more robust broader impact statement would be beneficial.

Questions

How do models which are fine-tuned compare to Gorilla? (If such a comparison is not appropriate, why?) What is the statistical significance of the results? What are the most common kinds of errors? Do the kinds of errors differ between the different categories of APIs? As a final minor question: do the authors have an explanation for why GPT-3.5 appears to outperform GPT-4 across a few different problem settings? This seems like a surprising and somewhat notable result!

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

I feel that the authors should have a more robust discussion of the broader impact of their work.

Summary

This paper introduces Gorilla, a fine-tuned LLaMA model designed to improve large language models' ability to use APIs accurately. The authors created APIBench, a comprehensive dataset of ML APIs from HuggingFace, TorchHub, and TensorFlow Hub, and used self-instruct to generate instruction-API pairs for training. They fine-tuned LLaMA-7B with retrieval-aware training, incorporating tech like AST sub-tree matching for evaluating API call accuracy, retriever-aware training to adapt to API changes, and handling of constrained API calls. Results show that Gorilla outperforms existing LLMs (including GPT-4) on API call accuracy across multiple datasets, demonstrates ability to adapt to test-time changes in API documentation, and handles constrained API calls effectively. The paper presents a novel approach to improving LLMs' API usage capabilities, with promising results that outperform existing state-of-the-art models in this specific domain.

Strengths

* Gorilla outperforms existing state-of-the-art language models, including GPT-4 and Claude, in API call accuracy across multiple datasets (TorchHub, HuggingFace, and TensorFlow Hub). * Gorilla significantly reduces API argument hallucination errors compared to other models, improving the reliability of API calls. * The retriever-aware training enables Gorilla to adapt to test-time changes in API documentation, allowing it to remain up-to-date with evolving APIs without requiring retraining. * Gorilla demonstrates the ability to understand and respect constraints (e.g., accuracy requirements) when selecting appropriate APIs, outperforming other models in constraint-aware API invocations.

Weaknesses

A significant weakness of the Gorilla approach is that it relies heavily on knowledge augmentation, which is a common technique already used in various domains to improve language model performance on specific tasks. The use of retrieval-augmented generation to enhance API calling capabilities doesn't represent a novel improvement or implementation compared to similar approaches in other domains. Essentially, Gorilla applies existing knowledge augmentation techniques to the specific task of API invocation, rather than introducing a fundamentally new method for improving language model capabilities. While the results show improvements in API calling accuracy, the core approach of combining retrieval with language model fine-tuning is not innovative in itself. This limits the broader impact and generalizability of the work beyond the specific domain of API invocation. The lack of significant methodological innovation suggests that similar performance improvements could potentially be achieved by applying existing retrieval-augmented generation techniques to the API calling task, without necessarily requiring the specific Gorilla architecture. This raises questions about the uniqueness and broader applicability of the Gorilla approach beyond the narrow domain explored in the paper.* While Gorilla performs well on the specific API datasets it was trained on, it's unclear how well it would generalize to entirely new APIs or domains not covered in the training data. Other Notes: * Although Gorilla shows some ability to handle constraints, its performance in this area is not significantly better than other models like GPT-3.5, suggesting room for improvement. * The paper doesn't compare Gorilla's performance with specialized API documentation tools or code completion systems, which might be more tailored for this specific task. * The paper doesn't provide an in-depth analysis of the cases where Gorilla fails, which could provide insights into its limitations and areas for improvement.

Questions

* How does Gorilla's performance compare to other specialized code generation or API-focused models, not just general-purpose LLMs? * What is the performance impact of varying the size of the training dataset or the base model size (e.g. using LLaMA-13B instead of 7B)? * How does Gorilla perform on more complex multi-step API workflows rather than just single API calls?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

1

Limitations

NA - Appropriately discussed limitations.

Summary

This paper addresses a pipeline to call adequate APIs among massive pools to accomplish users’ instructions. For that, the authors construct and release the APIBench dataset that contains more than 1645 APIs, and propose the Gorilla model, which is a retrieval-aware finetuned Llama-7B model for API calls.

Strengths

- Constructing the APIBench with AST tree matching evaluation metrics as well as open-sourcing the trained Gorilla model will be largely benefit to the community. - The problem in this paper is timely in terms of LLM applications and eco-systems — i.e., automatizing API function calls. - The fine-tuned model, Gorilla, surpasses performance compared to current SOTA models. - The paper is well written and soundly provides experiment results and analysis.

Weaknesses

- Lack of details about the training data for Gorilla, such as data stats and construction methods. It’s unclear how different the Gorilla training data and evaluation sets of APIBench are, which could lead to doubts about data contamination. - Experiment results analysis - How robust are models to paraphrased user instructions corresponding to the same target model API? - Providing performance for each domain could make it possible to analyze which domains are easy and hard. - Clarifications are needed: - In the experimental result section, the authors report two metrics along with overall accuracy: **the error by hallucination and by selecting the wrong API call.** The metric equations or explanations for calculating the values need to be clarified. - In section AST as a Hallucination Metric (line 251), how was experimented to attain the human evaluation of 78% accuracy? - It is required to mention the license of crawled APIs for usage. - A limitation section is absent, though the authors describe them in the checklist. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.

Questions

- In section 4.1, “Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. *However, for HuggingFace, since the dataset is not exhaustive, for all the models except Gorilla, we only check if they can provide the correct domain names.*” is vague. The numbers of data for HF, TFH, and TH are 925, 801, and 95, respectively. Please clarify this. - Table 1 shows that TorchHub is hard to even with oracle API documents compared to HuggingFace and TensorFlowHub. Can you draw the reason? - (Typos) line 225. `his suggests` - (suggestions) Please elaborate more on Table 2 — e.g., the caption and highlights. Moreover, it’s hard to connect between the written explanation and the table, due to different numbers. - (suggestions) Absence of mentioning the full name of AST (Abstract Syntax Tree)

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

- Limitation section is absent, though the authors describe them in the check list. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.

Response to the author

Thank you for your reply. Your explanation has mostly addressed my concerns. I hope you can incorporate some of the above content into the revised paper, such as the contributions and impact to the community, comparisons with other methods based on LLMs for tool calling, and some grammar issues. However, since there are still some experimental phenomena whose causes have not been fully explained, I suggest the author conduct a more in-depth analysis and investigation. Therefore, I have decided to keep my scores.