Toolformer: Language Models Can Teach Themselves to Use Tools

Summary

The paper explores an interesting area to extend large language models (LLMs) with external tools. The authors show that LLMs can teach themselves to better utilize tools. They tested GPT-j on several tools (calculator, QA system, search engine, translator, and calendar). The experimental results well support the claim, and the model even surpasses GPT-3 despite owning far fewer parameters.

Strengths

+ The paper is well-written and easy to follow. + The idea is novel and the supporting experimental results are extensive. + The authors study a very interesting topic which I believe will be impactful in the LLM era.

Weaknesses

The experiments are only conducted on GPT-j (a non-instruction tuned model), which I believe is not enough, considering the existence of more powerful open-source LLMs such as LLaMA and Vicuna. I actually tested that Vicuna, ChatGPT, and GPT-4 already have excellent capabilities in utilizing the tools mentioned in the paper (almost perfect). These models can skillfully manipulate tools based on very simple prompting, achieving far better performance than the reported number in this paper (I'm not sure why this paper only includes GPT-3 as the baseline, which apparently performs poorer than most of the current LLMs). Hence I doubt whether the proposed method could still benefit well-tuned SOTA LLMs.

Questions

NA

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

This paper proposes an approach to augment language models with the ability to call "tools" during decoding, such as a calculator, retrieval system, or machine translation system. This requires only a few human-written examples, and then uses the LM to generate a larger fine-tuning datasets constructed from raw text. When fine-tuned on this dataset, and augmented with the ability to execute external tools, performance of the LM is improved for a range of downstream tasks, across various model scales.

Strengths

* The paper proposes a relatively elegant way to integrate tools with language models in a way that requires only a limited amount of human-written examples of API calls per tool. The proposed method to synthetically construct the fine-tuning dataset appears to work well in practice. * By showcasing a variety of tools and their impact across a collection of tasks, this paper showcases the potential impact of integrating such tools and their ability to address some common limitations of LMs. The paper seems likely to influence future work.

Weaknesses

* I did not find any significant weaknesses in the proposed approach, execution of the experiments, or technical descriptions in the paper. * My only gripe is in the wording of the title claim that LMs can "teach themselves to use tools". I can see what the authors mean, as a LM is used to generate the fine-tuning data, but I don't find this to be a helpful description of the method and I think the paper would read better without this bit of hype. Additionally, for new tools, the approach still requires a prompt, a handful of examples, and heuristics for selecting relevant subsets of a corpus. Anyways, this gripe shouldn't be blocking for publication, and I don't expect the authors to change their selected title.

Questions

* The use of a threshold based on the likelihood assigned by the LM with and without the tool use is clever, but I also wonder whether this could be misleading in some cases. For instance, the LM may have been trained (?) on some of the CCNet data, so this may lead to an overly optimistic likelihood without tool usage relative to the optimal tool usage at test time, especially for, .e.g., temporally-sensitive facts. Were any such limitations related to the filtering process observed in practice?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Yes

Summary

This paper proposes an innovative method for enabling Language Models (LMs) to utilize tools. The authors prompt the LM to generate API calls based on human demonstrations, which are then executed in tools. Any non-contributing API calls are filtered out. A dataset is then augmented with these API calls, and used to fine-tune the LM. The Toolformer surpasses larger models in many tasks, offering a significant contribution to the field.

Strengths

This paper outlines a remarkably simple yet effective strategy for curating a dataset that empowers LMs to utilize tools. The method is well-explained and detailed, boasting a universal applicability across multiple datasets and tools. The authors have carried out extensive, well-designed experiments that showcase the performance boost facilitated by their method. The comparison experiment involving Toolformer, a disabled Toolformer, and GPT-J+CC is particularly commendable, as it eliminates the potential of additional fine-tuning data contributing to performance improvement. This research addresses a practical and intriguing topic that is likely to attract considerable interest from both the research and industrial communities. The potential to integrate more sophisticated tools and utilize larger LMs holds promise for advancing LM capabilities.

Weaknesses

The proposed method has some limitations. First, there's a dependency on fine-tuning when adapting the LM to new tools, which could impede broad usage and necessitate additional work. Secondly, the use of square brackets for the "<API>" token, without any special escaping mechanism, might present issues when square brackets form part of the original text. Lastly, the MLQA experiment raises a few questions. The performance of OPT(66B) and GPT-3(175B) suffers due to their inability to provide answers in English, suggesting a potentially inappropriate evaluation setting. Toolformer also underperforms GPT-J in certain languages, seemingly due to the impact of fine-tuning on CCNet. However, it's unclear why Toolformer lags behind GPT-J+CC in German and Arabic. The MLQA experiment fails to convincingly support the paper's main claims. Additionally, the paper lacks an analysis of cases where the LM fails to use tools effectively during inference. For instance, the reasons behind the LM's failure when using a calculator during inference are not investigated. Is it due to the inability to generate the <API> token or the candidate? Or does it fail to provide the correct answers even when the API call is successful? An examination of these failed cases and the issues causing them would be enlightening.

Questions

To summarize the inquiries raised in the Weaknesses section: Does the use of square brackets as API tokens interfere with the standard usage of square brackets in the text? Why does Toolformer underperform in comparison to GPT-J+CC in German and Arabic in the MLQA experiment? Under what circumstances does Toolformer fail to utilize tools effectively?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a method to finetune pretrained autoregressive language models such that they learn when and how to use external tools to achieve good performance in downstream tasks. Following an in-context learning scheme, humans provide a few examples of inserting API calls at appropriate location in the natural language sentence, which is ised by the trained language model to automatically generate an API-augmented variation of a natural language corpus. These API augmented sentences are filtered via a threshold over a criterion measuring whether adding the API calls and their result as a prefix improves the perplexity of the natural language sentence under the LM. This filtered API augmented dataset is used for further finetuning of the pretrained LM so that it learns to insert API calls at appropriate places. This approach called toolformer is compared against using the LM alone, finetuning the LM on the natural language corpus, using toolformer but aritificially suppressing its ability to call an API, and larger general purpose language models. 5 different APIs are considered in this setting. The model is evaluated on several downstream tasks for which access to the APIs might be beneficial.

Strengths

-- The paper is very well motivated. This capability of querying external API while generating text is a natural solution to the pathologies like hallucination that the language models exhibit today. This approach is a step toward endowing a language model with such capabilities reliably. -- The experimental setup is well designed and the choices of APIs, baselines, and downstream tasks to evaluate on lead to informative analysis. -- This approach outperforms baselines convincingly on the downstream tasks while not drastically affecting the language modeling capabilities as measured by perplexity on the held-out set.

Weaknesses

-- From the writeup, this approach doesn't seem to generate multiple API calls in a sentence and also doesn't perform well with nested API calls. More discussion on this would be useful. -- While it is discussed in the limitation section and Table 2, a more thorough analysis of sample efficiency of this approach would be helpful. How many sentences are enough to learn tool-use functionality? Is it possible to collect enough high-quality API augmented sentences easily? -- Ablation study: performance as a function of filtering threshold that controls the quality of the API-augmented sentences would give more insight into the learnability of tool-use and sensitivity to the "correctness" of the API-augmented dataset. -- A thorough error analysis of failure modes would improve the understanding of the limitations of the proposed approach more clearly.

Questions

See above

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above

Thanks for the response

I am keeping my initial score.