Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner. The project is available at https://chameleon-llm.github.io.
Paper
References (72)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This work presents Chameleon, a system that uses a library of tools or modules to assist a LLM to answer questions. First, a LLM is prompted with a description of available modules, constraints on how the modules should be used, few-shot demonstration examples, and finally the question to answer. The LLM then generates a sequence of modules that should be used to process the question. The modules are used in sequence, keeping track of module outputs and changes to the input question. Finally, the last module outputs an answer to the original question. When Chameleon uses GPT-4 as the base LLM, it achieves +2.55% improvement over GPT-4 with Chain-of-Thought prompting on ScienceQA, and +1.85% improvement over GPT-4 with Program-of-Thought prompting on TabMWP.
Strengths
I see 4 main strengths of this paper: 1. The idea of performing tool use entirely within a sequence of modules (instead of interleaving tool use with prediction from the original LLM) is new and offers some specific advantages 2. Achieving a small performance increase on two question-answering datasets 3. The project being an example of how engineering effort can be organized to achieve high dataset-specific performance 4. The paper writing is well organized and easy to read Originality: The most original aspect of this work is that the solution is constructed entirely within the modules, whereas previous work on tool use (to my knowledge) interleaves modules with prediction from the original LLM. With the Chameleon approach, the necessary reasoning at each step can be more tightly controlled (e.g., with module-specific prompts and heuristics about what information to include or exclude for future modules), and constraints can be placed on the sequence of modules to use. Quality: The authors provide an extensive comparison to prior models on two datasets, along with an ablation study for the importance of various modules. The experiments show a slight improvement over previous approaches. Clarity: The paper is well organized, and the supplemental material contains extensive additional details for the modules. Significance: The research topic (how to best use LLMs with tools) is definitely significant.
Weaknesses
Overall, this paper feels less scientific and more like a report on an engineering accomplishment in achieving high accuracy on the benchmark datasets, since the modules were created with the datasets in mind, a dataset-specific subset of modules is available at test time, each module includes hand-designed heuristics for how to update the input and cached information, there are hand-designed constraints on module ordering, and dataset-specific “default” module sequences are used if the predicted one does not meet the constraints. All of this points to my interpretation that Chameleon is a framework that can be instantiated, with *sufficient engineering effort*, into a system with high *dataset-specific* accuracy. However, there is little evidence that the Chameleon approach can produce a *single, general-purpose* question answering system with high accuracy on a variety of datasets. From my perspective, the main weakness is the lack of some general takeaway that can be applied to other scenarios. There are some avenues toward potential takeaways but currently they are not explored thoroughly enough to draw confident conclusions. In the Questions section below, I provide some potential takeaways that could be explored in more detail. Another weakness of the paper is that the writing oversells the approach in various ways. Please see the Questions section below for specifics. I hope the authors edit the writing accordingly. Originality: There is not much novelty other than the “sequence of modules” approach mentioned in the Strengths section. Although, if the “sequence of modules” approach is shown to be better than prior approaches through further experiments, then its novelty would be sufficient. Quality: It is unclear what the main takeaways of the paper should be, or how well-supported they are. Clarity: The writing oversells the work in a few ways. Significance: Some of the potential takeaways, if explored more thoroughly, could be quite significant. However, the actual conclusions one can draw from evidence in the paper are less significant right now.
Questions
## Potential takeaways that could be explored in more detail * Potential takeaway 1: performing tool use entirely within the sequence of modules is better than interleaving tool use with LLM generation as in prior work, e.g., ToolFormer. This might be due to constraints in module ordering, heuristics in how modules update the context, using module-specific prompts instead of the original prompt which might be overloaded, or interleaved tool use “interrupting” the flow of text generation. I would love to see an analysis of these different factors using the same set of tools, for a more general understanding of the benefit of Chameleon’s “sequence of modules” approach. * Potential takeaway 2: the Chameleon framework is truly general-purpose, meaning the same system achieves high performance on a variety of datasets. This would represent an important step forward in question answering. But currently, we only see dataset-specific variations of Chameleon performing well on one dataset at a time, which is less impressive because the comparison is to GPT-4 with CoT or PoT which has no dataset-specific design or engineering (other than a few-shot prompt). How does Chameleon perform on each dataset with all modules available, a single setting of module ordering constraints, and a single default module sequence? (If the performance is then worse than GPT-4 CoT or PoT, then is Chameleon truly a generally applicable approach?) * Potential takeaway 3: if the goal is to achieve high accuracy on a single academic dataset, then one can find success using Chameleon with dataset-specific engineering and heuristics. This takeaway is already supported by the paper, but in my opinion, this is a weak result. There are no dataset-specific approaches in the comparison (at least no recent ones building upon ChatGPT or GPT-4). These are also *academic* datasets that are meant to serve as a proxy for measuring more general ability; raw performance on these datasets without generalization to other tasks is not as impressive. On the other hand, if Chameleon could get high performance on a more general and broad dataset, for example (hypothetically) a representative subset of queries to a search engine or AI assistant that are related to science/math/reasoning/etc., then it could be argued that Chameleon is general enough for practical applications, and then this takeaway would carry more weight. In light of these, I'm not sure what readers are supposed to take away from the paper in its current state, and so I cannot recommend acceptance at this time. However I do think any of the potential takeaways above, if explored thoroughly, could lead to a much more impactful paper, so I encourage the authors to continue their investigations. ## Writing that oversells the approach * The abstract mentions “improving the best published few-shot result by 11.37%” on ScienceQA and 17.0% on TabMWP, and these numbers are repeated throughout the text. While technically true, these numbers are misleading, simply because the performance of GPT-4 with CoT or PoT hasn’t been published before. Certainly the authors can’t claim any “ownership” of GPT-4 plus CoT or PoT, so they should be treated as prior work even if the authors are the first to measure their accuracy on these datasets. Considering this, it is more honest to say that Chameleon achieves **+2.55% on ScienceQA and +1.85% on TabMWP** over previous approaches without fine-tuning. (MM-COT Large is a fine-tuned model achieving +5.14% over Chameleon on ScienceQA, according to Table 3.) Lines 65-68 are carefully worded to avoid mentioning those bolded numbers and should be more transparent. * Throughout the text, Chameleon is said to produce “NL-like programs”. Is “*programs*” the most accurate term to use here? These are merely **sequences of modules**, without any other programming feature like loops, conditionals, variable binding, helper functions, and so on. I think the term “program” is overselling the complexity of the module sequence plans. * What does “plug-and-play” mean? This is a key term used in the paper’s title, but I actually think it hides complexity. To add a new module, one must: (1) implement the core module logic, (2) describe the module in the planner’s prompt, (3) describe constraints about when to use the module with respect to other modules in the planner’s prompt, (4) implement checks for those constraints so invalid plans can be rejected, (5) provide few-shot demonstrations of using the module in the planner’s prompt, (6) decide for which datasets the module should be available, (7) design the module’s update_input and update_cache functions. This seems like quite a bit of work -- an acceptable amount of work, but not quite “plug-and-play”. If easily adding new modules is a key feature of Chameleon, then at the very least, the authors should clearly describe all the steps involved in using a new module, but this is currently missing from the paper. * Why is MM-COT Large not included in Figure 3(a)? It would be misleading to omit the best approach from the “main baselines”. * Some modules are always used in a particular order. From Figure 6 and 7 from the appendix, the following modules are always used together: (Query Generator -> Bing Search), (Solution Generator -> Answer Generator), and (Program Generator -> Program Verifier -> Program Executor). Would it make sense to combine each group into a single module? Table 1 lists Chameleon as having >10 tools, but if these tools were combined, the count would be 9. Chameleon is described as using modules in a compositional way, but the number of tools is inflated compared to which compositions are actually used. ## Minor questions and suggestions * Throughout the text: “**our** Chameleon” is awkward phrasing. It would be more natural to say “our approach Chameleon” or simply “Chameleon” * Line 97: correcting grammar - “has made tremendous progress and **has stimulated** research in **prompt** learning ...” * Line 125: extraneous space after Chameleon * Line 153: please provide examples of update_input and update_cache, since they are hand-designed for each module * Line 153: Is $M_t$ actually $M_i$? Or $M^t$? * Line 171 and so on: I didn’t notice any usage of the notation $M_{kr}$ elsewhere, is it necessary? * Line 233: missing quote before Knowledge Retrieval * Lines 246 - 248: this sentence is repeated * Line 251: “in the appendix” is repeated * Line 296: does the “solution-executor module” mean the “Solution Generator module”? --- Updated scores after the author-reviewer discussions: * Soundness: 3 -> 4 * Contribution: 2 -> 3 * Rating ("overall score"): 3 -> 4
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
2 fair
Contribution
3 good
Limitations
The limitations (Appendix B) should mention that it takes more wallclock time and compute resources to process questions with modules, especially if those modules themselves contain LLM components. If the use of multiple LLM-based tools really proliferates, then this would be a broader societal impact (Appendix C) through an increase in the amount of energy required to process queries, leading to greater environmental impact.
Responses to writing improvement suggestions
Thank you for pointing out the valuable suggestions for our paper. We will incorporate them in our revised paper: ### Result presentation Our intention was to highlight the improvements of Chameleon over the benchmarks **available at the time**. We understand that this may have created confusion. We appreciate the insight that these comparisons should be treated as prior work. We'll modify our description to state that Chameleon (GPT-4) achieves +2.55% on ScienceQA and +1.85% on TabMWP over previous approaches without fine-tuning. Additionally, we will make adjustments in Lines 65-68 to provide a more straightforward representation of the data. ### NL-like programs Our intention in using this term was to highlight that the generated sequences resemble the structure of a Python list, and its simplicity in natural language, aligning with the tool's name. We appreciate your insight and will consider using a more precise term such as “module sequence plans”, or provide an explanation to avoid confusion. ### Plug-and-play We agree that the process of adding a new module involves several steps, as you have outlined and we have discussed in the readme file of the codes. The term was intended to highlight the idea that, a new module can be easily integrated within the existing framework without needing to modify other components, compared to existing works. We appreciate your suggestion and will provide a section outlining the steps in the revision, thereby providing clarity and transparency about the process. ### MM-COT Large MM-COT Large boasts a much larger tuned parameter size of 738M, while the dominant fine-tune models only have tuned parameter sizes from 1.2M to 223M. We did include MM-COT Large in Table 3 for a comprehensive comparison, but it was excluded in Figure 3(a) to focus on comparisons within the dominant settings. Your feedback is valuable, and in the revision, we will include MM-COT Large in Figure 3(a) or provide a clear explanation. ### Modules in order tend to be used together due to the sequential nature of the tasks they perform. However, maintaining them as distinct modules has specific advantages: 1. **Flexibility and extensibility**: By maintaining module separation, each can be individually updated, improved, or replaced without affecting the others. For example, Solution Generator could be updated with fine-tuned language models, while Answer Generator can adapt to varied formats in new tasks. This modularity is pivotal for future refinements or extensions of Chameleon. 2. **Clarity in functionality**: Each module has a unique role in the overall reasoning process. Keeping them separate aids in understanding, debugging, and modifying, especially for those unfamiliar with our system's details. ### Limitation statement We acknowledge the extended time and compute resources required when using modules with LLM components. This insight will be detailed in Appendix B. Furthermore, the broader societal implications, especially the environmental impact of increased energy consumption, are crucial. We'll address these concerns and potential mitigations in Appendix C.
Thank you for the detailed rebuttal. However, I still feel that the main takeaway is unclear and not well supported. I don't think this can be resolved with only a rebuttal; I believe new experiments are needed to fully support whatever takeaway the authors wish to communicate. For example, Potential Takeaway 1 was: "performing tool use entirely within the sequence of modules is better than interleaving tool use with LLM generation as in prior work, e.g., ToolFormer". I wrote down a few potential reasons this might be, and suggested the authors perform "an analysis of these different factors", e.g., detailed ablation studies to measure the impact of each potential reason. The rebuttal elaborates on the reasons with words, without any deeper analysis. Potential Takeaway 2 ("the Chameleon framework is truly general-purpose, meaning the same system achieves high performance on a variety of datasets") might take less work to support with experimental results. The authors claim that, with GPT-4 recently supporting a context window of 16K tokens, Chameleon can support all of its modules in a dataset-agnostic way, and the authors also mention new positive results on FinQA and NER. However, no experimental results are provided to verify whether Chameleon **maintains improved performance on all datasets when set up in a dataset-agnostic way**. This is the crucial missing experiment that would support Potential Takeaway 2. I would like to clarify that I don't think it's necessary for one paper to explore _all_ of the potential takeaways. It is more important that _at least one takeaway_ is clearly communicated and very well supported with experimental evidence. But given the current results, I don't think any of those potential takeaways are sufficiently supported yet, so I encourage the authors to continue the work with a focus on demonstrating (at least) one message very well. My scores are unchanged, because the main weakness (lack of a clear well-supported takeaway message) remains.
Takeaways we claimed in the paper
Dear Reviewer, Thank you so much for your insightful comments and continued feedback. We would like to clarify **the takeaways we have claimed in our submitted paper, backed by extensive experimental results**. **Our takeaway 1: Augmented LLMs demonstrate flexibility and superior performance in complex reasoning tasks.** Our approach offers a distinct novelty. It not only decomposes tool planning and execution but also features a modular implementation of the framework. This offers advantages in scalability, adaptability, and ease of integration. Different from concurrent works that focus on runnable demos, we are among the pioneering efforts that examine their performance benefits in compositional reasoning contexts, as recognized by Reviewer yhWh. The performance benefits are eviendenced by comparison with latest few-shot and fine-tuned models, listed in Table 3 and 4. **Our takeaway 2: GPT-4 exhibits a more precise and consistent tool planning capability than ChatGPT.** We have investigated GPT-4's planning capabilities, emphasizing its superiority over ChatGPT. These findings are supported by experimental results showcased in Figures 4-7 and Table 7. **Our takeaway 3: Our additional research illuminates potential directions for future developments in LLM agents and augmented LLMs.** The ablation study in Table 5 shows the significance of different modules for complex reasoning tasks. Transition graphs in Figures 6 and 7 study the planning behavior of LLMs on program planning. Through examples presented in Figures 1-2, 8-9, Tables 19-24, and Figure 10 (added in [pdf](https://openreview.net/attachment?id=bIr8g5rvPm&name=pdf) during the rebuttal period), we demonstrate how Chameleon facilitates complex reasoning and identify scenarios where it might fail, providing a roadmap for future research. Warm regards, The authors of Paper 2615
Responses to your remaining concerns on potential takeaways
Dear Reviewer, Thank you so much for your great efforts in reviewing our paper. We highly appreciate your new feedback and would like to address them as follows: We would like to clarify that **we didn’t claim these three potential takeaways as you pointed out in our paper submission**. However, we genuinely appreciate the insightfulness of these potential takeaways and would like to delve deeper into them with **both existing and new experiments**. ### Potential takeaway 1 **In our submitted paper**, we **did not claim** that “performing tool use entirely within the sequence of modules is better than interleaving tool use with LLM generation as in prior work, e.g., ToolFormer”. As noted in the ToolFormer paper, ToolFormer generates over 300K data to fine-tune the LLM and only calls one API for every query. This approach is not suitable for complex reasoning problems that necessitate the combination of several different tools, especially when no extensive training data is available. Moreover, it lacks efficiency; for instance, when calling N APIs, the output from the first API would be redundantly fed to the LLM N-1 times. This paradigm, which interleaves tool use with LLM generation, has not been favored in subsequent works as shown in Table 1. Given theses limitations and inefficiencies, a direct experimental comparison between ToolFormer and Chameleon is not currently practical. **We genuinely hope for your understanding on this matter and seek further guidance on whether you'd still like us to conduct such experiments.** ### Potential takeaway 2: the Chameleon framework is truly general-purpose We're grateful for your suggestion and have implemented Chameleon in a **dataset-general** manner, as you recommended. Specifically, when assessing various datasets, both the planner and tool inventory remain consistent, including task description, tool set, and demonstrations. As evidenced by the results presented below, **Chameleon, when deployed in a dataset-general fashion, maintains similar performance improvements over current SOTA baselines**. New results on 1,000 test examples of TabMWP: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 90.9 | / | | GPT-4 PoT | dataset-customized | 96.9 | / | | Chameleon (ours) | dataset-customized | **98.8** | **+1.9%** | | Chameleon (ours) | **dataset-general** | **98.5** (new) | **+1.7%** (new) | New results on 1,000 test examples of ScienceQA: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 82.6 | / | | Chameleon (ours) | dataset-customized | **84.9** | **+2.3%** | | Chameleon (ours) | **dataset-general** | **84.8** (new) | **+2.2%** (new) | Additionally, it is a common optimization strategy to employ task-specific and query-specific adaptations to enhance both performance and API-calling efficiency when developing few-shot LLMs. This approach is supported by multiple papers as follows: - Learning to retrieve prompts for in-context learning, NAACL22 - Automatic Chain of Thought Prompting in Large Language Models, ICLR23 - Complexity-Based Prompting for Multi-Step Reasoning, ICLR23 - Eeast-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR23 ### Potential takeaway 3: Chameleon can achieve high performance on general datasets As the results below indicate, Chameleon achieves an improvement ranging from 1.85% to 11.85% on **four challenging reasoning datasets** compared to strong GPT-4 based baselines. It is therefore evident that Chameleon has the potential to be generalized to broader datasets. | | **ScienceQA** | **TabMWP** | **FinQA (New)** | **NER (New)** | | --------- | ------------------ | ------------------ | ------------------- | ------------------- | | GPT-4 CoT | 83.99 | 90.81 | / | / | | GPT-4 PoT | / | 96.93 | 62.87 | 56.71 | | Chameleon | **86.54 (+2.55%)** | **98.78 (+1.85%)** | **74.72 (+11.85%)** | **68.10 (+11.39%)** | We hope that these clarifications, combined with literature references and additional experimental evidence, validate our takeaways and address your concerns about potential takeaways. **We will incorporate all these clarifications and new experiments into our revised paper.** We are deeply grateful for your invaluable comments and continued feedback, which have been instrumental in enhancing our paper! Warm regards, The authors of Paper 2615
New results with Claude and the main takeaway
Dear Reviewer, We have some new results regarding our Chameleon approach for the updates. We switched the base LLMs to **Claude** and observed similar performance improvements. Our proposed method achieves an accuracy gain of 2.1% on ScienceQA and a gain of 15.7% on TabMWP, compared to SOTA few-shot approaches such as CoT and PoT. The new results using **Claude** as the base LLM: | | ScienceQA | TabMWP | | - | - | - | | CoT (new) | 78.7 | 74.0 | | PoT (new) | / | 79.6 | | **Chameleon** (new) | 80.8 **(+2.1%)** | 94.9 **(+15.7%)** | The results using **ChatGPT** as base LLM: | | ScienceQA (%) | TabMWP (%) | NER (F1) | | - | - | - | - | | CoT | 76.9 | 83.3 | / | | PoT | / | 89.3 | 51.3 | | **Chameleon** | 78.9 **(+2.0%)** | 94.2 **(+5.1%)** | 68.1 **(+16.8)** | The results using **GPT-4** as base LLM: | | ScienceQA (%) | TabMWP (%) | FinQA (%) | NER (F1) | | - | - | - | - | - | | CoT | 82.6 | 90.9 | / | / | | PoT | / | 96.9 | 62.9 | 56.7 | | **Chameleon** | 84.9 **(+2.3%)** | 98.8 **(+1.9%)** | 74.7 **(+11.8%)** | 66.7 **(+11.0)** | The **main takeaway from our paper** is that **augmented LLMs using the Chameleon framework can demonstrate flexibility and superior performance in complex compositional reasoning tasks**. This is validated through extensive experiments and analysis, as follows: 1. Experiments on TabMWP and SciencenQA using ChatGPT and GPT-4 as base LLMs, and comparisons with SOTA approaches (Table 1 and Table 2). Our proposed method achieves a 1.9%-5.1% improvement. 2. New experiments on two financial reasoning benchmarks, FinQA and NER, show similar substantial improvements with an accuracy gain of 1.9%-5.1% and an F1 score gain of 11.0-16.8, achieved by our method. 3. New experiments using Claude as the base LLM. Our approach demonstrates consistent improvements across two tasks. 4. New error analysis in [pdf](https://openreview.net/attachment?id=bIr8g5rvPm&name=pdf) indicates that our proposed method significantly reduces the errors made by base LLMs due to their inherent limitations in using external tools. 5. The ablation study presented in Table 5 underscores the importance of tools with specific functions for complex reasoning tasks. 6. The study of tool-use planning (Figure 4 and 5) reveals that GPT-4 possesses superior tool-planning reasoning compared to ChatGPT. 7. Quantitative studies in Table 7, and Figures 6 and 7 show that the LLM planner can effectively decide how to sequence tools in a few-shot setup. We sincerely hope that our new experiments and clarifications address your remaining concerns. Thank you very much for your consideration! Best regards, The authors of Paper 2615
The new results are very nice. I am glad to see that the improvements in the initial paper were not due to dataset-specific choices, since the improvements remain mostly the same after making the approach dataset-general. This is strengthened by positive results on new benchmarks and demonstrating improvements using different base LLMs. --- I want to be really clear: through my emphasis on "takeaways", I am _not_ directly asking the authors to perform any particular experiment; instead, I am identifying _potential_ takeaways and how those might be supported, in an effort to help the authors strengthen the paper's message. It is up to the authors to choose what key message(s) their paper should communicate. I only want to make sure that those takeaways are clearly communicated and well-supported with evidence. I'd like to briefly comment on the takeaways that the authors do claim: > Our takeaway 1: Augmented LLMs demonstrate flexibility and superior performance in complex reasoning tasks. In my opinion, this is just saying "tool use is powerful" which is well-known at this point. This is not really a new discovery and thus doesn't carry much significance. > Our approach offers a distinct novelty. It not only decomposes tool planning and execution but also features a modular implementation of the framework. Do the authors want to highlight, as a _key contribution_, their software engineering design which makes the system easily extendable? If so, more details about the engineering would be needed. What clever design decisions were made and why? Practical evidence such as "a colleague who did not contribute to the original code base took X amount of time and Y lines of code changes to add a new module" would also help. > Our takeaway 2: GPT-4 exhibits a more precise and consistent tool planning capability than ChatGPT. I'm really not surprised because GPT-4 is much better than ChatGPT in many ways. Like "Our takeaway 1", I don't think this one carries much significance either. > Our takeaway 3: Our additional research illuminates potential directions for future developments in LLM agents and augmented LLMs. It was not clear to me how this paper illuminates clever new research directions that were not known before. > The main takeaway from our paper is that augmented LLMs using the Chameleon framework can demonstrate flexibility and superior performance in complex compositional reasoning tasks. In my words, "Chameleon is dataset-general and performs well with several base LLMs". Considering the new results, I think this is now the strongest takeaway and is well supported. I am quite happy about this. --- Due to the new results, I am updating my scores. My reasoning is: * From my initial review, **the main weakness has been remedied**. Specifically, "Chameleon is dataset-general and performs well with several base LLMs" is a clear takeaway message and is supported with the new results. * My **new main concern** is simply about the writing being updated. The authors have worked hard to perform new experiments with (1) a new dataset-general setup, (2) new base LLMs, and (3) new datasets. These new experiments are _crucial_ to supporting the paper's message (i.e., they aren't just going into an appendix). Certainly, the paper's main text requires a significant amount of updating: * Updating the description of the method which used to be dataset-specific but is now dataset-general * Providing background on the new base LLMs * Providing background on the new datasets, and discussing/analyzing prior works that targeted those datasets * General reframing or clarification of the paper's message * Any other revisions/clarifications according to all of the author-reviewer discussions * Who will review all of the updated text which in total would seem to be a "major revision"? It is unfortunate that the authors are not allowed to upload a revised paper during the discussion period. **I am hesitant to recommend acceptance knowing that the paper will have major changes that are completely unreviewed** (and I think back my concerns about the authors' initial writing, as mentioned in my initial review). Initial scores: * Soundness: 3 good * Presentation: 2 fair * Contribution: 2 fair * Rating: 3 reject Updated: * Soundness: 4 excellent * Presentation: 2 fair (I expect this would go up if I were to review a revised paper) * Contribution: 3 good * Rating: 4 borderline reject (I expect this would go up if I were to review a revised paper) To the authors: regardless of the accept/reject decision, the work is now _much improved_, and I hope the authors feel that the review process has been helpful.
Appreciation for Your Helpful Feedback Throughout the Rebuttal
Dear Reviewer uov2, We are sincerely grateful for your invaluable comments and suggestions throughout the reviewing and rebuttal process. It's encouraging to know that our new experiments have addressed your concerns regarding the generality of Chameleon. **We will incorporate all the new findings into our revised paper and refine it based on the writing suggestions from you and other reviewers**. We believe these updates will make our paper more impactful and appeal to a broader audience in the field. Although we can't update the paper at this moment, **we will release it publicly, and both you and the community are welcome to provide further feedback on our work**. The rebuttal process has been very helpful and fruitful for us! Thank you so much for your significant contribution to our paper. Best regards, The authors of Paper 2615
Thanks for Your Valuable Reviews: Kind Request for Reevaluation
Dear Reviewer, Firstly, we wish to express our gratitude for your thorough review of our paper. We genuinely appreciate the time and expertise you dedicated to its evaluation. We have taken each of your comments to heart and addressed them in our rebuttal. It would be immensely beneficial if you could reconsider our revised arguments and clarifications. We are hopeful that our revisions have addressed your concerns satisfactorily and might warrant **a reevaluation of its acceptance**. Your feedback has been, and continues to be, pivotal to our work. We would highly value any additional insights or clarifications you might offer. Thank you once more for your time and consideration. Sincerely, The authors of Paper 2615
Summary
The paper presents Chameleon, an AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. Chameleon uses various tools, including LLMs, vision models, web search engines, and Python functions, to perform complex reasoning tasks. The system, powered by GPT-4, significantly improves performance on two multi-modal reasoning tasks: ScienceQA and TabMWP, setting new state-of-the-art results. The authors also demonstrate that Chameleon's GPT-4-powered planner exhibits more consistent and rational tool selection compared to a ChatGPT-powered planner.
Strengths
1. The paper is overall well-written and easy to follow. Abundant figures and visualizations also help with reader understanding. 2. The paper presents a novel system, Chameleon, that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. This innovative approach addresses the inherent limitations of LLMs, such as their inability to access up-to-date information, use external tools, or perform precise mathematical and logical reasoning. 3. Chameleon achieves impressive performances (especially with GPT-4) on two QA datasets: ScienceQA and TabMWP, showing the effectiveness of the entire framework. 4. Very detailed and thorough analysis of existing related works, which can largely help the community to explore the direction.
Weaknesses
1. I might miss something. However, the novelty of the method seems to be limited. As a method focusing on compositional tool use, I am wondering what are the differences between Chameleon and HuggingGPT (https://arxiv.org/pdf/2303.17580.pdf), except for the application tasks and involved tools. As both works seem to leverage human-introduced ordering information between tools to compositionally use the tools. 2. Both datasets seem to have strong dependencies between different tools. With a relatively small number of simple tools (compared with vision models and SQL interpreters, etc.), the compositional order among tools seems to be quite static and rigid.
Questions
Overall a good paper. No additional questions.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have thoroughly demonstrated the limitations of the work in the appendix.
Summary
This work proposes Chameleon, an approach to connect large language models to different tools for compositional reasoning. Chameleon is powered by a GPT-4 planner that chains different tools to accomplish the described goal. Chameleon achieve improved performance in comparison with strong baseline models.
Strengths
* The presentation of the framework is great. The paper has nice figures, clear definitions of problems, clear demonstrations of results. * The experiments are solid. This work evaluates Chameleon on two challenging benchmarks with distinct purposes * The study on how to make LLMs such as GPT-4 plan reasonably is useful for future research.
Weaknesses
* The improvement on the two benchmarks is not as significant as described in the abstract. The abstract indicates that "on ScienceQA, improving the best published few-shot result by 11.37%" where the best-published result is a GPT-3 based approach while Chameleon is powered by GPT-4. Therefore, the comparison is not completely fair. The improvement does not necessarily come from the benefit of Chameleon. Instead, It could be as simple as GPT-4 being more powerful (as many tools are prompt-based). In Table 3, Chameleon (GPT-4) only shows a 2.55% improvement over GPT-4 CoT, and 1.62% when comparing Chameleon (GPT-3.5) with GPT-3.5 CoT. * The paper does not explain *how* to apply Chameleon to other tasks. Even though Chameleon is claimed to be plug-and-play, there are many *task-specific* designs. For instance, the `Row_Lookup`, `Column_Lookup` , `Table_Verbalizer` tools are highly specialized for table-related questions, and they are very fine-grained. How does a practitioner know whether to have both `Row_Lookup` and `Column_Lookup`, or a simple `Table_Lookup` tool? What are the insights on tool selections? Do we need a dev set to "fine-tune" the tool pool? These details need to be further explained to make Chameleon really plug-and-play. * Another concern is that converting the output of one tool (module) to fit the input format of another tool is pre-defined (L153-L157), indicating that it is potentially challenging to generalize Chameleon to *massive* number of tools due to the extensive human efforts.
Questions
* How does Chameleon select the module inventory? What is the rule of thumb? * How are `update_input` and `update_cache` are implemented? Any examples? * Prompt-based tools such as `Row_Lookup` are powered by the same model as the planner; is that correct?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
* Be more accurate in the improvement presentation
A Sincere Request for Further Feedback
Dear Reviewer, We sincerely thank you for the time and expertise you've dedicated to reviewing our paper. Your feedback has been instrumental, and we've carefully addressed each point in our rebuttal. Considering the clarifications we've provided, we kindly ask that you revisit our responses. We're hopeful that these adjustments align with your expertise and could move the paper's standing closer to *a stronger acceptance*. Your continued feedback is pivotal for us. Thank you so much! Sincerely, The authors of Paper 2615
Reply
Thanks for the response, I appreciate the detailed answers. The requirement of a **domain expert** confirms my (and reviewer yhWh) concern about the "plug-and-play" feature of Chameleon. Overall, Chameleon provides a guideline for solving *specific* tasks. I would remain the same score. As a side note, what seems to be helpful to me is an easy-to-use software that encodes the design philosophy proposed in Chameleon so that domain experts could quickly implement pipelines for their tasks. Unfortunately, the code folder is empty in the supplementary material.
Responses to your remaining concerns
Dear Reviewer, We deeply appreciate your efforts in reviewing our paper. We highly value your new feedback and would like to address them as follows: We believe that empowering domain experts to furnish tools that assist LLMs is a cornerstone for enhancing human-AI collaboration in tackling intricate tasks. **The goal of our framework is not to build a general AI that can automatically perform every task, but to offer a practical solution that facilitates the easy adaptation of LLMs to deliver high-quality performance using existing modules**. This is especially beneficial for domain experts who might not have extensive knowledge in adapting machine learning models. This approach has not only proven practical but has also attracted wide interest, as evidenced by concurrent works listed in Table 1. The novelty and significance of our research are underscored by Reviewer yhWh, who described it as “one of the first works in the field,” and Reviewer GU39, who noted its potential to “largely help the community to explore the direction”. We wish to emphasize that **the human effort is, in fact, notably minimal compared to other alternatives that inject expert opinions**. **The only inputs expected from domain experts are**: (1) a concise natural language description outlining the task's objectives; (2) a set of tools necessary for this task; and (3) a few straightforward demonstrations of how the task employs these tools in responding to user queries. The **“plug-and-play”** feature of Chameleon facilitates the inclusion of new tools via additional descriptions and demonstrations. It also allows for the seamless updating and replacement of existing tools. Chameleon has the advantages of simplicity, adaptability, and efficiency over concurrent works. For example, Gorilla-UCB demands the assembly of tool APIs guided by human expertise, the generation of 16K instruction data, and subsequent LLM fine-tuning. HuggingGPT focuses on devising runnable demos, rather than validating the performance advantages of LLM agents, an endeavor we undertook. Reviewer yhWh further acknowledged these merits, describing them as **“straightforward but necessary for LLM-based agents”**. We have indeed included the code in the Supplementary Material during the paper submission. **You could access our codes via the following link**: [https://openreview.net/attachment?id=HtqnVSCj3q&name=supplementary_material](https://openreview.net/attachment?id=HtqnVSCj3q&name=supplementary_material). Our codes are well-structured, complemented by detailed instructions for execution. We have made all result files available to aid researchers in reproducing our findings and gleaning insights for upcoming research. Warm regards, The authors
Thanks
Thanks to the authors for the update. I briefly walked through the code base and checked the latest response to reviewer uov202, I raised my score from 6 to 7 for the following reasons: * Chameleon can be *relatively* general on tasks with similar components (e.g., require using a table). I appreciate the authors also added new experiments on NER and FinQA. * The associate code base can be helpful for domain experts who have specific tasks in mind. Although currently, the code seems to be organized w.r.t datasets, I think it is a matter of time to repurpose the code to serve broader tasks. For instance, the code could provide APIs to allow users to easily define their tools and their tasks. It will be nice if the authors could provide more **design guidelines** for new tasks. For instance, how could we know whether to have both `Row_Lookup` and `Column_Lookup`, or a simple `Table_Lookup` tool?
Thanks for your acknowledgment and the raised score
Dear Reviewer, We sincerely appreciate your time spent reviewing our code and the responses to Reviewer uov2. We are encouraged by your acknowledgment of our work and your raising the score to 7. We are pleased to hear that you appreciated the general applicability of Chameleon and that our additional experiments addressed your concerns. **These discussions will be included in our revised paper**. Regarding the code base, we recognize that its current structure was organized to enhance simplicity and readability. However, as you pointed out, it allows for straightforward repurposing for broader tasks, given our modular approach. Similar components can be reused across different tasks. **We are actively modifying the code base to make it organized more generally**, as you suggested. Your idea of providing APIs to allow users to easily define their tools and tasks is very insightful. It is vital for both industry applications and research challenges. **We will discuss this idea in our revised paper**, as we believe it offers significant value for researchers and practitioners in this field. Furthermore, **we plan to delve deeper into this valuable idea in our future work** and explore how the expertise of domain experts can be naturally integrated into AI-assistant LLMs for improved human-computer interaction. Regarding your suggestion of providing design guidelines for new tasks, we completely agree. **We will offer comprehensive, hands-on documentation complete with demo examples in our code base.** This will include step-by-step instructions along with the corresponding code implementations for developing new example tasks. We believe this documentation will assist users in adapting Chameleon for new tasks based on our existing code. To determine if a specific tool is needed, **the current strategy** is to provide all available tools with their descriptions to the system. Our qualitative study in Section 5.2 shows that LLMs like GPT-4 can deduce a new composition of tools even without a direct demonstration within the LLM prompt. **A more innovative solution** would be to categorize the tools based on specific skills. In such a setup, users would only need to identify the relevant skill category, and the system would then automatically retrieve associated tools tailored for the task at hand. For example, if the user identifies the need for the table understanding skill, the system would furnish all related tools, such as `Row_Lookup`, `Column_Lookup`, and `Table_Lookup`. The agent itself is able to compose suitable tools for a specific query. **We will discuss the idea of skill-centered LLM agents in our revised paper and are eager to explore it in our future work**. Once again, thank you immensely for your valuable insights and continuous input on our paper! Best regards, The authors
Summary
The paper introduces Chameleon, a novel AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. The authors argue that while LLMs have achieved remarkable progress in solving various natural language processing tasks, they have inherent limitations in accessing up-to-date information, using external tools, and performing precise mathematical and logical reasoning. To address these drawbacks, Chameleon synthesizes programs by composing various tools, such as LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules, for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. The authors showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. ScienceQA requires the system to answer science-related questions by integrating information from text, tables, and images, while TabMWP involves generating natural language descriptions of tables based on input queries. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. The authors also provide a detailed analysis of Chameleon's performance on various sub-tasks and demonstrate its ability to handle complex queries that require multiple tools and intermediate steps. They further discuss the potential of Chameleon for addressing real-world queries across various domains and improving the state of the art in multi-modal knowledge-intensive reasoning tasks. The paper concludes by highlighting the importance of enhancing current LLMs with the capability to automatically compose external tools for real-world task solving.
Strengths
1. This paper probably is one of the first works in the field that explore automatic task planning and execution with GPT-4 as the agent. It further enhances the abilities of GPT-4 by leveraging various tools controlled by a GPT-4 based planner. Such a combination is straightforward but necessary for LLM-based agents. 2. One merit I like about this paper is that it has clear evaluation metrics, not like other agent-based models that merely have a runnable model and use cases. The standard evaluation clearly helps us understand the position of this paper and the performance of the proposed method. 3. The improvement on top of GPT-4/ChatgPT is significant and clearly shows the effectiveness of the proposed method. As we can see from Tables 3 and 4, the proposed method can mostly improve their direct baselines, i.e., ChatGPT and GPT-4. Also, qualitative analyses are helpful in understanding the model behavior.
Weaknesses
1. Like the other recent works for LLM agents, the novelty of such kind of works is somehow ad-hoc and limited. There are many tool-augmented works, as demonstrated in Table 1, and their differences are petty nuanced, and it's hard to tell the real novelty behind them. Usually, each such kind of work would leverage the most recent LLMs, like GPT-4, to do the planning or task decomposition and call special tools to solve each subtask. Most powers come from OpenAI's model, esp. GPT-4, which is hardly claimed to be the credit of these works (but it's good to see the proposed methods can further improve GPT-4). This leaves us with an essential question, what are the core contributions of such works? Demonstrating GPT-4 can be augmented by external tools definitely is nontrivial but also engineering-heavy. For NeurIPs works, I may like to see more research values; for example, which designs of the proposed method really distinguish itself from other similar works? Is it the "generating a program in a natural-language-like format"? Or some special prompt format to trigger the planning and task execution abilities? etc etc. 2. Although the paper names its method Chameleon, trying to highlight the adaptability and versatility of LLMs, ironically, the task scope and experiment settings in this paper are limited. The planning pipeline and proposed tools are specially designed for solving the two datasets. While conceptually, the idea of augmenting GPT-4 with some tools can definitely be applied to more general tasks, this generality comes from this simple idea but not from the proposed method. I hope the authors could dedicate more effort to make the proposed method easier to be applied to more tasks. 3. Another weakness of the proposed method probably is the simple planning design, which is basically a straightforward CoT with a linear structure. As we can see from Table 3, there is still a large gap between fine-tuned STOA and the proposed method. One possible reason might be the weak planning ability of the proposed method. This is somehow related to the first weakness, meaning that most LLM agent works combine GPT-4 with tools in a very simple way. I hope the authors could explore more advanced planning algorithms to see if the performance can be further improved.
Questions
I raised many questions in the weakness section, and the authors may want to resolve them and clarify some questions. The following is some minor comments: 1. I don't quite get the connection between the name "Chameleon" and the proposed method. I also don't quite get the meaning of the colors in the name. 2. I'd suggest reducing the table space, removing most of the unnecessary baselines. The only meaningful baselines probably are the STOA from previous works (mostly fine-tuned?) and GPT-4/ChatGPT. 3. Considering some improvements on top of ChatGPT and GPT-4 are kind of marginal, I'd encourage the authors to conduct a thorough error analysis to figure out how the base ChatGPT/GPT-4 make mistakes and how the proposed method reduces such errors.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
N/A
Kind request for your continued feedback
Dear Reviewer, Thank you very much for your time and efforts in reviewing our paper. We truly value your insights and have diligently addressed them in our rebuttal. As we greatly esteem your expertise, we kindly request that you take a moment to review our responses. We're eager to address any additional feedback, questions, or clarifications you might have. We're hopeful that our responses may *prompt a reconsideration of the paper's rating*. Your continued feedback is crucial to us. Sincerely, The authors of Paper 2615
Request for acknowledgment our responses and new experiments
Dear Reviewer, We have some updates relevant to your comments with three new experiments. **We hope that these new results and planned revisions will further underscore the value of our work, encouraging you to possibly adjust and elevate your rating**. This is immensely important to us. Thank you! ### Experiment 1: The performance of Chameleon in a dataset-general way We added experiments on Chameleon implemented in a dataset-general manner, as suggested by Reviewer uov2. In this setup, when assessing different datasets, both the planner and tool inventory remain consistent. The results below indicate that **Chameleon, when deployed in a dataset-general fashion, maintains similar performance gains over SOTA baselines**. New results on TabMWP: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 90.9 | / | | GPT-4 PoT | dataset-customized | 96.9 | / | | Chameleon (ours) | dataset-customized | 98.8 | +1.9% | | Chameleon (ours) | **dataset-general** | **98.5** (new) | **+1.7%** (new) | New results on ScienceQA: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 82.6 | / | | Chameleon (ours) | dataset-customized | 84.9 | +2.3% | | Chameleon (ours) | **dataset-general** | **84.8** (new) | **+2.2%** (new) | ### Experiment 2: The performance of Chameleon on general datasets | | **ScienceQA** | **TabMWP** | **FinQA (New)** | **NER (New)** | | --------- | ------------------ | ------------------ | ------------------- | ------------------- | | GPT-4 CoT | 83.99 | 90.81 | / | / | | GPT-4 PoT | / | 96.93 | 62.87 | 56.71 | | Chameleon | **86.54 (+2.55%)** | **98.78 (+1.85%)** | **74.72 (+11.85%)** | **68.10 (+11.39%)** | ### Experiment 3: Error analysis of ChatGPT and Chameleon Thanks for your insightful suggestion. We have added an error analysis comparing ChatGPT and Chameleon. The tools in Chameleon, designed for image captioning and knowledge retrieval, notably reduce errors in these areas. Sequential tool execution further lessens errors in solution generation. GPT-4's task planning significantly outperforms ChatGPT. A detailed discussion can be found in [pdf](https://openreview.net/attachment?id=bIr8g5rvPm&name=pdf). Error categories and numbers: | | **All** | **Tool Planning** | **Image** | **Knowledge** | **Solution** | | ------------------- | :-----: | :---------------: | :-------: | :-----------: | :----------: | | ChatGPT | 50 | / | 32 | 37 | 27 | | Chameleon (ChatGPT) | 35 | 13 | 10 | 6 | 17 | | Chameleon (GPT-4) | **28** | **1** | **19** | **3** | **8** | ### Planned revisions for our final paper In our final paper, we plan to: - **Add new experiment results and discussions**; - **Add clarifications** Chameleon's novelty and its adaptability to diverse datasets. - **Add the discussion of future work**, such as introducing APIs that allow users to define their tools and tasks, and developing skill-centered LLM agents, as suggested by Reviewer 1xpZ. - **Modify the code base** to enhance its organization. - **Add comprehensive documentation**, supplemented with demo examples in our code base, assisting users in adapting Chameleon for new tasks. Best regards, The authors of Paper 2615
Kind Request for Discussions and Feedback for Paper 2615
Dear Reviewers, Firstly, we'd like to express our sincere gratitude for taking the time to review our paper and for providing invaluable feedback. We have made our best efforts to address all the comments raised during the initial review. If you've had a chance to go through our responses, we kindly request your feedback. If our responses have satisfactorily addressed your concerns, we hope you might consider adjusting your ratings to reflect these changes. Your engagement during this discussion period is crucial for the improvement of our work. We genuinely value your insights and look forward to your continued feedback. Thank you for your time and consideration. Warm regards, Authors of Paper 2615
Thanks for the response
Thanks the authors for their detailed rebuttal. I have no further questions and I will maintain my positive score.
Thanks for your positive feedback
Dear Reviewer, Thank you very much for your thoughtful review and your positive assessment of our work. We appreciate the time and effort you have invested in this process, and we're glad to hear that our rebuttal has addressed your questions satisfactorily. Should you have any further questions in the future, or if there are any additional aspects that need clarification, please do not hesitate to reach out to us. Sincerely, The authors of Paper 2615
Kind request for review of our new experiments and clarifications
Dear Reviewer, We greatly appreciate the time and effort you dedicated to reviewing our paper and offering insightful comments. During the rebuttal period, we added **multiple new experiments**. We kindly draw your attention to these updates and eagerly anticipate any further feedback. Your insights are invaluable to refining our paper! ### Experiment 1: The performance of Chameleon in a dataset-general way We added experiments on Chameleon implemented in a dataset-general manner, as suggested by Reviewer uov2. In this setup, when assessing different datasets, both the planner and tool inventory remain consistent, including task description, tool set, and demonstrations. The results below indicate that **Chameleon, when deployed in a dataset-general fashion, maintains similar performance gains over SOTA baselines**. New results on TabMWP: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 90.9 | / | | GPT-4 PoT | dataset-customized | 96.9 | / | | Chameleon (ours) | dataset-customized | 98.8 | +1.9% | | Chameleon (ours) | **dataset-general** | **98.5** (new) | **+1.7%** (new) | New results on ScienceQA: | | **Implementation** | **Accuracy (%)** | **Gain over SOTA** | | ---------------- | ---------------------- | -------------------- | ---------------------- | | GPT-4 CoT | dataset-customized | 82.6 | / | | Chameleon (ours) | dataset-customized | 84.9 | +2.3% | | Chameleon (ours) | **dataset-general** | **84.8** (new) | **+2.2%** (new) | ### Experiment 2: The performance of Chameleon on general datasets | | **ScienceQA** | **TabMWP** | **FinQA (New)** | **NER (New)** | | --------- | ------------------ | ------------------ | ------------------- | ------------------- | | GPT-4 CoT | 83.99 | 90.81 | / | / | | GPT-4 PoT | / | 96.93 | 62.87 | 56.71 | | Chameleon | **86.54 (+2.55%)** | **98.78 (+1.85%)** | **74.72 (+11.85%)** | **68.10 (+11.39%)** | ### Experiment 3: Error analysis of ChatGPT and Chameleon We added an error analysis comparing ChatGPT and Chameleon, as suggested by Reviewer yhWh. The tools in Chameleon, designed for image captioning and knowledge retrieval, notably reduce errors in these areas. Sequential tool execution further lessens errors in solution generation. GPT-4's task planning significantly outperforms ChatGPT. A detailed discussion can be found in [pdf](https://openreview.net/attachment?id=bIr8g5rvPm&name=pdf). Error categories and numbers | | **All** | **Tool Planning** | **Image** | **Knowledge** | **Solution** | | ------------------- | :-----: | :---------------: | :-------: | :-----------: | :----------: | | ChatGPT | 50 | / | 32 | 37 | 27 | | Chameleon (ChatGPT) | 35 | 13 | 10 | 6 | 17 | | Chameleon (GPT-4) | 28 | 1 | 19 | 3 | 8 | ### Planned revisions for our final paper In our final paper, we plan to: - **Add new experiment results and discussions**; - **Add clarifications** Chameleon's novelty and its adaptability to diverse datasets. - **Add the discussion of future work**, such as introducing APIs that allow users to define their tools and tasks, and developing skill-centered LLM agents, as suggested by Reviewer 1xpZ. - **Modify the code base** to enhance its organization. - **Add comprehensive documentation**, supplemented with demo examples in our code base, assisting users in adapting Chameleon for new tasks. We deeply value your feedback on our paper. **We sincerely hope that these new results and planned revisions heighten your appreciation of our work, prompting you to reconsider and potentially elevate your score**. This is immensely significant to our paper. Thank you! Best regards, The authors of Paper 2615
Thanks for the rebuttal
I appreciate the details responses and additional experiments from the authors. My score will remain the same for the following reasons: 1. The arguments from the rebuttal against my concern about the novelty don't add new and convincing information for me. Being the first batch of works in a new field doesn't mean its weakness in novelty can be neglected to some degree. The current design is more like directly applying GPT-4 to solve some datasets intuitively and I believe the authors can do better, for example, I point out that the planning and more sophisticated/general design could be potentially promising directions for improved novelty. 2. The versatility of the proposed method isn't fully justified, even with the new results on "dataset-general" implementation. The whole paper is tailored in the sense that the tools, modules, and designs are specifically designed for two datasets (two other datasets are added in the rebuttal but the details are unclear and I don't think it will change the whole flow of the paper). I think this is directly against the intention behind the name Chameleon. 3. I really appreciate the new error analysis from the authors and I believe it should be included in the next version. However, there is one interesting observation that GPT-4 dramatically improves the tool planning over ChatGPT, which very likely contributes to the final improvement significantly. This implies that it is GPT-4 that plays one of the most significant roles in this pipeline, which isn't surprising somehow but also leaves the question that which should be attributed to the essential success of this kind of framework, OpenAI or the authors? This question probably won't influence the acceptance of this paper (also OpenAI probably has received enough credits) but I feel should raise some certain level of concern for the community. 4. Echoing reviewer uov2 somehow, I do feel the main takeaway or positioning of this paper isn't super clear to me and the overall writing oversells the contributions a little bit. When the follow-up works refer to `Chameleon`, do we expect a general framework elegantly combining GPT-4 and tools, or a variant of Auto-GPT, or a github project to solve a few specific datasets? Considering the divided opinions among other reviewers, regardless of whether this paper will be accepted or not, I do hope the authors can further think of the above questions (considering restructuring the paper, method, and experiments to make the work really stand out, I do like the LLM+tool idea though) and incorporate them into the revised version. Essentially, being the first batch of works doesn't mean too much, and it is the core contributions that last longer than the influence of a paper's acceptance. Thanks.
Responses and new results with Claude
We deeply appreciate your continued feedback. In response, we’d like to make clarifications with **new experiments** and hope they address your concerns. ### Novelty of our work We wish to underscore the **scientific contribution** of our work. Concurrent research studies the potential of LLMs by merely implementing runnable demos, as pointed out by you. Therefore, the extensive experiments and analysis explored in our research should not be overlooked. Advanced approaches depend on extensive computational resources and a lengthy context window, however, which were **not feasible due to the techinque limitations during our research period**. LLMs like GPT-4 were only available to a limited audience and had a maximum context window of 4K tokens. Fortunately, thanks to recent advances, LLMs have become more accessible to the broader public. The context window has expanded to 16K for GPT-4 and an impressive 100K for Claude. We are keen to delve into advanced planning designs as you've proposed. For instance, how can LLM agents refine their tool planning strategies during task execution? ### Versatility of the proposed method The versatility of our proposed method stands out when compared with concurrent works in Table 1: 1. It **supports tools of various types and skills**. Our method uses natural language to describe tools and store interaction information. In contrast, works like WebGPT depend on domain-specific programs. 2. It generates **query-specific** tool compositions, whereas works such as MM-REACT reply on **task-specific** tool usage. 3. Its **modular design** enables both the planner and each tool to be updated or replaced individually. This differs from systems like Toolformer, which require collecting additional large-scale training data and fine-tuning the entire system. New experiments executed in a limited timeframe during the rebuttal phase further indicates its ease of development and adaptability to new datasets and settings. We will further clarify these points and reorganize the paper to provide a more coherent presentation of our method and experiments. ### Error analysis and final improvement Thank you for acknowledging our error analysis; it will be included into our revised paper. Regarding the final improvement from our proposed method, we'd like to clarify that **a fair comparison would be assessing the performance gain of Chameleon over SOTA approaches, given the same base LLM**. The base LLM primarily influences the foundational performance of various LLM approaches, but not the final improvement of our method. We've demonstrated our method's improvement on three different LLMs, with **Claude being added** during the rebuttal: The improvement using **Claude** as the base LLM: | | ScienceQA | TabMWP | | - | - | - | | Claude CoT (new) | 78.7 | 74.0 | | Claude PoT (new) | / | 79.6 | | **Chameleon (Claude)** (new) | 80.8 **(+2.1%)** | 94.9 **(+15.7%)** | The improvement using **ChatGPT** as base LLM: | | ScienceQA (%) | TabMWP (%) | NER (F1) | | - | - | - | - | | ChatGPT CoT | 76.9 | 83.3 | / | | ChatGPT PoT | / | 89.3 | 51.3 | | **Chameleon (ChatGPT)** | 78.9 **(+2.0%)** | 94.2 **(+5.1%)** | 68.1 **(+16.8)** | The improvement using **GPT-4** as base LLM: | | ScienceQA (%) | TabMWP (%) | FinQA (%) | NER (F1) | | - | - | - | - | - | | GPT-4 CoT | 82.6 | 90.9 | / | / | | GPT-4 PoT | / | 96.9 | 62.9 | 56.7 | | **Chameleon (GPT-4)** | 84.9 **(+2.3%)** | 98.8 **(+1.9%)** | 74.7 **(+11.8%)** | 66.7 **(+11.0)** | We have observed that, compared to SOTA approaches such as CoT and PoT, our Chameleon approach consistently achieves improvements across different base LLMs. For instance, there is a 1.9%-15.7% accuracy gain and an 11.0-16.8 F1 score gain. The improvement with Claude (developed by Anthropic, not OpenAI) underscores the **generality** of the Chameleon framework. This indicates that **base LLMs can be updated and replaced independently**, and, if annotation data is available, they can also be updated with fine-tuned **open-sourced** models. ### The revised version We are encouraged by your appreciation of the LLM+tool idea. We genuinely hope that our work will be impactful in the field and provide insights for future work on augmented LLM and LLM agents. To this end, we will provide refine our paper, incorporating new experiments, clarifications, and discussions. We will reorganize our code base for greater generality and provide comprehensive documentation to allow users to adapt our approach to broader tasks.
Decision
Accept (poster)