Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AvaTaR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demonstrate AvaTaR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AvaTaR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar.
Paper
Similar papers
Peer review
Summary
This paper introduces a new framework, AVATAR, that allows LLM agents to optimize the performance of Knowledge Retrieval. In the framework, a Comparator agent is adopted to extract insight from the positive and negative samples. The experiments on four retrieval datasets show the effectiveness of the method.
Strengths
1. The schematics in the paper are well drawn and demonstrate the proposed method well. 2. The experimental settings and prompts are detailed. 3. The experiments are performed on STARK and Flickr30K Entities benchmarks to show the effectiveness of the methods.
Weaknesses
The main weakness is the lack of innovation and comparison compared with cutting-edge works. 1, This proposed AVATAR is very similar to ExpeL [1] in terms of method and has no obvious innovation in comparison. ExpeL adopts a Reflexion agent (actor) to gather success and failure experiences of multi-step tasks, which is the same as the actor's role in this paper. ExpeL adopts another agent to compare a failed trajectory with a successful trajectory for the same task and extract insights from the comparison, which is the same as the instruction generation process of the comparator in this paper. ExpeL adopts the experience pool to retrieve successful trajectories, which is the same as the Memory Bank in this paper. Similarly, Autoguide [2] also extracts insights from experiences. Therefore, the method of auto-optimizing the instruction by comparing the successful and failed trajectories has no innovation compared with Expel [1] and Autoguide [2]. The author should conduct more surveys and read relevant cutting-edge works. 2, The paper claims their method achieves SOTA on STARK and FLICKR30K-ENTITIES benchmarks, but they only compared with relatively basic methods. The author employs several outdated embedding-based retrievers as the comparison. However, text-to-image retrieval on FLICKR30K-ENTITIES has long been dominated by the vision-language model [3] [4]. Internvl [3] and Beit [4] achieve high recall@1 (from 80 to 90) on this benchmark. Therefore, it is unreasonable to claim that the proposed method achieves the retrieval SOTA on these two benchmarks. 3, The results of the comparison on FLICKR30K-ENTITIES are missing. Quantitative results on FLICKR30K-ENTITIES are only shown in Figure 5 (right), but the results of all methods on this benchmark are not found in the paper. Considering that STaRK is a relatively new benchmark, not many methods have experimented on it, so results and analysis on recognized benchmarks are important. 4, Many sentences in the paper are too obscure to understand and lack clear explanations. For example, "these per-sample instructions tend to be narrow and fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary with others, leading to decreased performance when applied across different scenarios." in lines 152-154, what do "per-sample instructions" mean, and why will these tool combinations "lead to decreased performance"? Providing some simple examples can make it easier to read. [1] Zhao, Andrew, et al. "Expel: Llm agents are experiential learners." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024. [2] Fu, Yao, et al. "Autoguide: Automated generation and selection of state-aware guidelines for large language model agents." arXiv preprint arXiv:2403.08978 (2024). [3] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [4] Wang, Wenhui, et al. "Image as a foreign language: Beit pretraining for vision and vision-language tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Questions
In line 160, "the comparator samples a group of data samples (question-answer pairs), executes the current actions for each question," but how can the comparator execute the current actions? What are the current actions for the comparator? The text in Figure 3 is too small to read and the meaning of the marked text is difficult to understand.
Rating
4
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
There is no separate Limitation or Broader Impacts section in the paper. In addition, the authors said, "We did an extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations," but the discussion of the limitations in this paper is required.
Thanks for the detailed responses, but they have not fully addressed my concerns. 1- Even though there are differences in details, the existence of papers like ExpeL and AutoGuide clearly reduces this paper's innovation in the direction of self-optimization of LLM agents. Given ExpeL and AVATAR both utilize the comparator to optimize the prompt, the main innovation of this paper would be replacing 'off-policy' with 'on-policy' learning. However, as 'off-policy' and 'on-policy' are both useful for RL, the authors should perform more experiments and studies to show that their 'on-policy' methods are better than the 'off-policy' version. 2- Previous works (ExpeL and Retroformer were published in Aug 2023) used ReAct and Reflexion as baselines, which cannot be the reason that this paper still uses the same baseline. The field of LLM Agents is a rapidly developing field. In the year after reflexion (published in Mar 2023), a large number of high-quality methods have been proposed, e.g., [1][2][3]. However, in the experiment, the author ignored the comparison with these cutting-edge methods. In addition, in the introduction and related work of the paper, the author repeatedly pointed out that the previous LLM Agent methods via self-optimizing cannot solve complex problems. However, there is no comparison with these methods in the experiment to verify these claims. Therefore, the authors should keep up with recent methods, and consider these approaches in their experiments, rather than just the basic methods. 3- The method proposed in this paper is not only applicable to retrieval tasks but also to general agent tasks. Therefore, it should also be experimented on some common and difficult benchmarks. As I said before, one of my concerns is that STARK is a very new benchmark (published in April 2024), and Flickr30k-entities is a very rarely used benchmark for LLM Agent. Achieving good results on LLM agent benchmarks that have been studied more, such as HotpotQA [4], AgentBench [5], and WebArena [6], the method can be more convincing. 4- For the question about the clarification of the comparator, your answer should be added to the revised version because the presentation of the original text is unclear and misleading. 5- One more new question: I found the authors implemented AVATAR with a batch size of 20: "the metric Recall@20 for constructing positive and negative queries, and hyperparameters (l = h = 0.5, b = 20)". Given that 20 trajectories of Actors can be quite long, can the authors provide the context length required by the comparator and the token overhead required for AVATAR optimization? [1] Zhu, Zhaocheng et al. “Large Language Models can Learn Rules.” ArXiv abs/2310.07064 (2023): n. pag. [2] Majumder, Bodhisattwa Prasad et al. “CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization.” ArXiv abs/2310.10134 (2023): n. pag. [3] Qian, Cheng et al. “Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution.” ArXiv abs/2401.13996 (2024): n. pag. [4] Yang, Zhilin et al. “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.” Conference on Empirical Methods in Natural Language Processing (2018). [5] Liu, Xiao et al. “AgentBench: Evaluating LLMs as Agents.” ArXiv abs/2308.03688 (2023): n. pag. [6] Zhou, Shuyan et al. “WebArena: A Realistic Web Environment for Building Autonomous Agents.” ArXiv abs/2307.13854 (2023): n. pag.
2nd Batch response
We appreciate the reviewer for further communication. We hope our response this time could mostly address your concerns. ### **TL;DR;** - We clarify Avatar's novelty, especially on AvaTaR $\neq$ ExpeL + on-policy training. - We show that AvaTaR outperforms ExpeL and Retroformer on HotpotQA. We also show the performance and scalability advantage of AvaTaR over ExpeL on STaRK-MAG; - We show that AvaTaR works well on new QA datasets (HotpotQA, ToolQA, ArxivQA) --- Before started, we make the following terms consistent to avoid confusion: - Queries = instances (used in AvaTaR) = tasks (used in ExpeL) - Actions to answer queries = action sequences (AvaTaR) = trajectories (ExpeL) --- **Comment 1: About novelty** While we pointed out "on-policy" and "off-policy" as one difference between AvaTaR and ExpeL, the novelty of AvaTaR goes beyond that. Here are our reasons: - **What to contrast**: This is relevant to the reviewer's question about batch size - *"Given that 20 trajectories of Actors can be quite long."* We believe this is a misunderstanding (please let us know if otherwise). Please note that AvaTaR contrasts positive and negative queries, unlike ExpeL, which contrasts trajectories. Therefore, the context length is less of an issue since the queries are mostly short, we will provide detailed statistics. - **Benefits of contrast queries**: Following the last point, the reviewer actually pointed out one benefit of contrasting queries - it allows us to contrast among a batch of pos/neg queries `(batch_size=b)` rather than a well-performing and an under-performing trajectory (`batch_size=2`). This design offers better scalability and generalization ability. We provided intuitions in L173-184, and we repeat them here for your convenience: ``` Moreover, as contrastive reasoning directly targets disentangling the performance gap related to input patterns and how they are handled differently by the tools, it is particularly effective in helping comparators differentiate and select tools for use. Finally, by identifying systemic flaws across a wide array of negative queries, comparator generates modifications that are not only tailored to individual samples but also to diverse data samples, offering benefits for better generalization to novel samples. ``` This design enables AvaTaR to generate more holistic instructions from the insights on multiple queries and/or generalize the final action sequence to hundreds of testing queries without test inference, which are validated by the strong generalization in our existing experiments. - **AvaTaR $\neq$ ExpeL + on-policy training**: Due to the differences in design, AvaTaR is able to conduct on-policy training since it directly optimizes the action sequence for multiple queries. However, one can imagine it would be hard for ExpeL to conduct on-policy training with trajectory comparison. --- **Comment 2: Comparison with more baselines** We added ExpeL and Retroformer as our baselines in our paper. We firstly compare them with AvaTaR on HotpotQA on the dev subset (100 queries) in their repositories. Here are the results, where all the baseline results are reported by ExpeL and Retroformer): ||HotpotQA (EM)| |:---|--:| |Act|29%| |ExpeL|39%| |ReAct |40% | |Reflexion|46%| |Retroformer (#retry=1)|51%| |AvaTaR|55%| For AvaTaR, we take 100 training queries to optimize prompts for 10 steps and set the number of retries as 0, which achieves the best performance. Then, we compare AvaTaR with ExpeL on STaRK-MAG. STaRK datasets involve over 20 tools, leading to long action sequences and high token overhead for ExpeL to process and evaluate. Due to this, we randomly sampled 100 training queries and evaluated ExpeL on 50 testing queries, comparing with the other methods on the same set. |||MAG|(#Test=50)|| |:---|:---|:---|:---|:---| ||Hit@1|Hit@5|Recall@20|MRR| |Dense Retriever|16.00|40.00|51.84|27.39| |QAGNN|20.00|52.00|49.71|36.39| |VSS|40.00 |58.00|55.93|47.76| |Multi-VSS|32.00|58.00|58.81|43.58| |ReAct|46.00|60.00|54.67|50.92| |ExpeL|40.00|58.00|55.94|47.43| |Reflexion|48.00| 64.00| 57.43|52.31| |AvaTaR-C|44.00|60.00|52.49|50.16| |AvaTaR|52.00| 64.00|53.86 |56.74| For ExpeL, we used `text-embedding-3-large` embedding model to retrieve insights. We found ExpeL’s performance similar to ReAct, which might be because the STaRK queries are diverse, therefore require more training data to gather enough experience. We will have ExpeL results on the other two STaRK datasets and include a table similar to Table 2 in our paper. Hopefully, these will address your concern about the comparison with previous LLM Agent methods and justify our claims.
2nd Batch response (Continue)
**Comment 3: More benchmarks.** Our experiments on HotpotQA improve this aspect. Please also see our response to `Reviewer CkbF` for AvaTaR results on ArxivQA and ToolQA. **Comment 4: Clarity on comparator.** Thanks! Yes, we made sure the statement is clear now. **Comment 5: Content requirements and token overhead.** On STaRK-MAG, the context length for the comparator is approximately 4k tokens per step, including initial instructions and tool usage (with pos/neg queries taking around 0.8k). The actor's token cost, including memories, is around 8k. Running AvaTaR for 50 steps accumulates about 600k tokens, costing under $10 with gpt-4-turbo. There is no inference cost on STaRK dataset as we directly apply the action sequence. As a reference, we also compute the token cost for ExpeL on STaRK-MAG: Training token cost: 357k in total (3.57k per task); Testing token cost: 389k in total (7.8k per task). In this comparison, we believe AvaTaR has an advantage in scaling especially when the number of testing queries are large. For HotpotQA, the comparator's context length is around 2k tokens per step (with pos/neg queries at 0.5k). The actor's token cost ranges from 2k to 8k, depending on the number of actions. Running 10 steps accumulates around 80k tokens. *Additional related works.** Thanks! We have also added them to our related work.
Summary
This paper introduces AVATAR, a novel framework for optimizing large language model (LLM) agents to effectively use provided tools and improve performance on complex multi-step tasks, with a focus on retrieval tasks. The key innovation is a comparator module that generates holistic instructions to improve the actor (main agent) through contrastive reasoning on batches of well-performing and poorly-performing queries. The authors demonstrate AVATAR's effectiveness on four challenging retrieval datasets from the STARK and FLICKR30K-ENTITIES benchmarks. Results show significant improvements over state-of-the-art baselines like ReAct and Reflexion.
Strengths
Novel approach: The comparator module using contrastive reasoning on query batches is an innovative way to generate holistic instructions for improving agent performance. This addresses limitations of per-sample instruction approaches. Clear motivation and explanation: The paper draws insightful analogies to concepts like batched training and gradient computation in neural networks to explain the intuition behind the approach. This makes the core ideas easy to understand. Strong empirical results: AVATAR consistently outperforms strong baselines across multiple datasets, with significant improvements on key metrics. The ablation study clearly demonstrates the value of the comparator module.
Weaknesses
Limited scope of evaluation: While the retrieval tasks are complex, it would be valuable to see AVATAR applied to a broader range of tool-use benchmarks to demonstrate generality. Lack of comparison to finetuning: The paper does not discuss or compare to alternative approaches like directly finetuning the actor model via rejection sampling. It's unclear how AVATAR compares to such methods in terms of performance and efficiency. Scalability considerations: The paper does not thoroughly address how the method scales with increasing numbers of tools or more complex task structures.
Questions
Regarding the use of the memory bank as in-context learning examples, How does AVATAR compare to directly finetuning the actor model in terms of performance, efficiency, and data requirements? What are the key advantages of the instruction-based approach? How does the method handle scenarios where the action model improves significantly, making it difficult to find negative examples for the contrastive learning process? Is there a strategy for addressing this? Have you explored applying AVATAR to other types of tool-use tasks beyond retrieval? What challenges do you anticipate in extending to more diverse task types?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have not adequately addressed the limitations of their work. I recommend adding a dedicated limitations and broader impacts section to address these points more thoroughly. This should include: Discussion of computational requirements and scalability limitations, potential failure modes or scenarios where the method may struggle
(Continue) Experiments on more tool-use benchmarks
Hi Reviwer CkbF, Thanks for waiting for the follow-up results on ToolQA! We introduce the setup and present the full results here: - **Datasets**: SciREX (a dataset for document-level information extraction based on full-length ML scientific papers) and Agenda (personal agenda questions based on a private knowledge base) are the two (and only two) datasets based on external text corpora from ToolQA. Each dataset has `easy` and `hard` splits. - **Configuration for AvaTaR:** We randomly split the questions by 40%:60% for training and testing, as official dataset splits are not provided. We set the maximum epoch to 5. - **Tools:** We use the knowledge base tools provided by ToolQA and web search tools for all agent methods. - **Agent Backbone:** We use `gpt-4o` for the LLMs of AvaTaR and the baselines, which are evaluated on our testing split for fair comparison. | | SciREX | SciREX | Agenda | Agenda | |------------|--------|--------|--------|--------| | | Easy | Hard | Easy | Hard | | CoT | $0.0$% | $0.0$% | $0.0$% | $0.0$% | | ReAct | $8.3$% | $18.3$% | $31.6$% | $11.7$% | | Reflexion | $10.0$% | $13.3$% | $30.0$% | $13.3$% | | AvaTaR | $11.6$% | $21.7$% | $36.7$% | $23.3$% | We observed consistent improvements across these two datasets, with a significant improvement on the hard queries of Agenda. We hope the experiments on ToolQA and ArxivQA can further validate the effectiveness and applicability of AvaTaR on more tool-use datasets. We are happy to provide any details (e.g., insights or example instructions) that are not covered here for conciseness.
Summary
This paper proposes a new framework AVATAR for LLM agent that operates in two stages: -The first stage is optimization during the training process, which integrates the LLM comparator component into the AVATAR. The comparator summarizes holistic prompts from positive and negative queries and iteratively optimizes LLM actor using the prompts. -The second stage is deployment during the inference process, where participants respond without the involvement of the comparator.
Strengths
Developing prompts for LLM agents is heuristic and laborious, but this paper proposes a new method called AVATAR, which automatically iteratively generates holistic prompts from positive and negative queries. Experimental results show that AVATAR outperforms SOTA methods. and some points summary as following: 1. It is a novel framework, comparing with the current SOTA approach such as ReAct, the Comparator component design is reasonable that it generates prompts from positive and negative queries. 2. It is new SOTA of LLM Agent. Experiments are conducted against current SOTA approaches as baseline, the framework outperform the current SOTA 3. It is a common framework, and it is effective with both Claude and GPT4.
Weaknesses
1. The optimization is data-driven, so comparator works effectively when the positive and negative queries are well sampled and balanced during iteration; when to stop iteration and how to sample the queries; it is not clearly clarified. 2. Regarding the construction of positive and negative queries, it is necessary to conduct experiments on the lower and upper bounds(l&h) to select appropriate values for optimal performance. 3. Regarding the memory bank, it is a important component to swap in and out according to data quality or balanced distribution, it is not well designed and discussed.
Questions
Is "AIR (gpt4)" in Table 3 a typo?
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
n/a
Correction: In the summary table above, the reviewer labels 'JqLP' and 'vMWD' should be switched, which matches the sequence order on the webpage. Apologies.
Decision
Accept (poster)