AGILE: A Novel Reinforcement Learning Framework of LLM Agents

Summary

This paper jointly introduces an evaluation environment for LLM agents and an LLM agent architecture called AGILE. The environments is a Question-Answering environment concerning the characteristics of commercial products sourced from Amazon. The benchmark itself is composed of 3 tasks: 1) a fact retrieval task where the agent should retrieve information about a product, 2) a product query task where the agent should retrieve a product that matches query criteria, and 3) a reasoning task where the agent should reason from the knowledge it has about a product to answer questions about possible uses or suitability of a product. The agent is composed of an LLM backbone that, under the control of an executive module, can retrieve memories of previous interactions stored in its database and use tools. Tools are used for agentic flow (reflect, submit answer), for memory management (retrieve memory, put new memory in database), and for asking advice from a human expert (access to ground truth about a product). The LLM is finetuned with SFT on these tasks. The agent is further fine tuned on a scored version of the tasks with reinforcement learning (PPO). The authors demonstrate the superiority of their architecture on their proposed environment and on a medical QA benchmark over one-shot prompting gpt4, and demonstrate that reflection, tool use, memory, ground truth product advice, and reinforcement learning contribute positively, with varying degrees, to agent performance.

Strengths

* This paper combines known LLM agent modules and studies their impact in two question-answering domains that are of interest to industry practitioners: product recommendation and medical knowledge; * Agents asking for help has been studied at length in the embodied question answering literature with previous-generation agents (see for instance https://arxiv.org/abs/1909.01871 or https://arxiv.org/abs/2302.04865), but as far as I am aware has not been studied with LLM-based agents; * The paper is fairly clear and easy to read, the proposed contributions are easy to understand, examples are provided for the task, and the results are comprehensive; * The claims of the paper are well substantiated with extensive experiments, and the ablations allow to easily disentangle the effect of different components of the agent; * The data collection process for creating the benchmark seems sound and the task looks challenging and interesting; * Reinforcement-Learning studies of complete LLM agents are rare and needed. * The tables in the related work makes the relationship of this work and other previous art clearer.

Weaknesses

* An overarching and pervasive weakness of this paper that is difficult to ignore and that has a great influence on my judgement is the claim of novelty of the agent architecture. In a post-Voyager (https://arxiv.org/abs/2305.16291) world, LLM-based agents equipped with tool use and retrieval are standard and the AGILE agent presented in this work introduces little conceptual novelty in the agent architecture, while even the title of the paper claims that the agent is novel. LLM agents of this type are so much part of the landscape that many reviews exist that organize existing approaches (see for instance the excellent https://arxiv.org/abs/2309.02427 that is not that recent either). The approach is standard enough that advanced software stacks (langchain, llamaindex) are devoted to speeding up building retrieval-augmented generation (RAG)-based agents for industry practitioners. The paper framing should be updated accordingly to reflect that the approach being studied is standard, but applied to a new task. * Relatedly, it would have been important for the authors to exactly pinpoint the difference between their agent and for instance Voyager, and either implement baselines from existing work on their task or test their agent on other tasks (eg WebShop which is very similar in spirit). * There are still novel elements in the proposed approach. The first is the use of reinforcement learning. While the use of reinforcement learning to improve language agents is not novel (see https://arxiv.org/abs/2302.02662 for a textworld task or https://arxiv.org/abs/2403.04642 for a math word task), the use of RL in RAG agents has not been studied as far as I am aware. It also makes up a significant portion of the method section and the appendix, however the experiments demonstrate only a marginal improvement due to RL. Why is this the case? How are the training curves like? Has the training converged? I also do not see any standard deviation information for the RL part. The authors justify the lack of error bars with the computational cost of experiments with is fair and completely acceptable where the comparison between methods is straightforward, but would have been helpful in the sft/ppo comparison where the difference is small and could have been due to chance, and it makes sense to concentrate computational effort on a truly novel part of the contribution; * The other novel element of this paper is the asking for advice part: the model can opt out of the task by seeing the ground truth which it can then report, at a cost (for the RL agent). We can see that allowing this action to the agent increases its accuracy, that increasing the cost of advice results in RL agents using advice less often, and that memoryless agents use advice more often. This is interesting, and could have been a core contribution of the paper, if it were explicit and investigated as such. The conditions under which LLM agents are aware of their own knowledge, as well as them taking appropriate steps to reduce lack of information is a research programme in itself, as has been discussed by John Schulman in his Berkeley lecture in 2023: https://www.youtube.com/watch?v=hhiLw5Q_UFg&t=215s Can the authors predict when the model is going to ask for advice? Does the RL training help the model know what it knows (metacognitive skills)? These are interesting questions that the authors could have investigated; * Relatedly, the related work paragraph on uncertainty looks unfinished. Relevant work on LLM’s metacognitive abilities are cited but there is no word on how the current work contributes to the discussion; * (minor) Time of training and number of gpus required for training should appear after the type of gpu; does the training use lora? * The evaluation for longform answers is based on GPT4, but no evidence is presented to correlate agreement of GPT4 with human judges on this task.

Questions

* My first and most important suggestion is that the authors should change the framing of the paper. The experimental work is sound and the tasks are worthwhile, but in the current state of the paper the conclusions seem to be that RAG agents work better than prompting on question-answering tasks, which is known. However, the paper could be: 1) a study on RL in RAG agent settings, 2) a study on training metacognitive abilities of RAG agents, and/or 3) a paper introducing a new benchmark (but the tasks seem not that challenging compared with tasks like SWE-bench https://www.swebench.com/ or reasoning tasks like theorem proving https://github.com/openai/miniF2F, https://leandojo.org/). * [Conclusion, l305] why is LLM system 1 and interaction with tools system 2? What does it mean? If the claim is to be left in the paper it should be explained and adequate citations should be provided (also is this analogy fruitful? What purpose does it serve? is this a claim about the cognitive plausibility of the proposed agent architecture or its resemblance to humans?). * [Clarity] The paper should explain what mcqa is when first mentioned, since it is confusing for people not familiar with the benchmark. Same goes for the meerkat model. * Why is the improvement from RL only 2%? * The original RAG paper should be cited: https://arxiv.org/abs/2005.11401

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

* Some limitations with respect to lack of resources are acknowledged in the appendix (small open source models are trained, only a subset of the training set is used). This is fine; the methods used are pretty powerful and can be expected to scale with increasing compute. * If understood as a way to help agents answer queries, expert advice is definitely non-scalable. It is fine as a research question, maybe less so for modelling real-world applications. This could be mitigated with long-term memory of the agent (give advice once, retrieve forever). In any case, a discussion on the feasibility of expert advice in real world scenarios would be nice.

Summary

This work presents a novel framework for LLM agents named AGILE. The entire AGILE system is trained end-to-end using reinforcement learning. A key feature of AGILE is its ability to seek advice from external human experts. Additionally, the authors have developed ProductQA, a challenging dataset of complex QA, to comprehensively evaluate the capabilities of the agent. Extensive experiments show that an agent within this framework, even when based on a smaller model and trained with RL, can outperform GPT-4.

Strengths

1. The paper introduces a novel framework, AGILE, for LLM agents that integrates memory, tools, and expert consultations, all optimized through reinforcement learning. 2. The development of the ProductQA dataset. 3. The paper is well-organized and clearly written. The architecture and workflow of the AGILE framework are explained with clarity. 4. The experiments on ProductQA and MedMCQA verify the AGILE framework and show that AGILE agents based on 13B and 7B LLMs trained with PPO can surpass GPT-4 agents.

Weaknesses

1. It is somewhat like WebGPT (in terms of actions and policy learning strategy), with extensions supporting memory, expert consultation, and reflection. 2. The experiments are all related to a single agent. It would be better to show the application for multiple agents and include planning.

Questions

1. How can this framework be extended to support multiple agents? 2. In Table 5, the accuracy of "w/o Memory" and "w/o Tool-Use" is higher than that of agile-vic13b-ppo. Is the metric "Accuracy – Advice Rate" a more straightforward one than Total Score?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitations.

Summary

This paper proposes AGILE, a reinforcement-learning based framework for finetuning LLMs for conversational QA tasks. The models are initially trained using imitation learning, then further finetuned with RL. Once finetuned, the models show strong performance, surpassing GPT-4 while using much smaller models.

Strengths

- The proposed framework is interesting and novel, distilling the usage of various tools, reflection, memory retrieval/writing, and human advice seeking from larger models' trajectories, to a smaller model, then further training it using RL to surpass the performance of the larger model - The performance is strong - Introduces a new dataset, ProductQA which is a useful resource for conversational QA in online shopping scenarios. - Cost-benefit analysis of advice seeking is insightful.

Weaknesses

I only have two main concerns: - Because the model is only evaluated on two conversational QA datasets (and not on any of the mentioned datasets in Table 8), it's difficult to judge the generality of the proposed method. - There is no comparison with standard ppo

Questions

What is the reason behind why there is not comparison with rl finetuning baseline (standard ppo)?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper "AGILE: A Novel Framework of LLM Agents" introduces a new framework for Large Language Model (LLM) agents designed to handle complex conversational tasks. The framework, named AGILE (AGent that Interacts and Learns from Environments), incorporates LLMs, memory, tools, and interactions with experts. AGILE is formulated as a reinforcement learning (RL) problem and is fine-tuned using Proximal Policy Optimization (PPO). The authors present a new dataset, ProductQA, for evaluating the framework and report that AGILE agents outperform GPT-4 in their experiments. The paper claims significant improvements in performance due to the integration of memory, tools, expert consultation, and RL training.

Strengths

**Comprehensive Evaluation:** The creation of the ProductQA dataset and the extensive experiments conducted on both ProductQA and MedMCQA provide a robust evaluation of the framework’s capabilities. **Significant Performance Improvements:** The reported improvements over GPT-4 agents in both ProductQA and MedMCQA are noteworthy, indicating the effectiveness of the AGILE framework. **Detailed Methodology:** The paper provides a thorough explanation of the RL formulation, training processes, and the roles of different components within the framework, which enhances reproducibility.

Weaknesses

**Limited Novelty:** The reliance on human experts for advice, while useful, is not a novel concept and has been explored in previous works and becomes a common practice. The paper does not sufficiently differentiate its approach to human expert interaction from existing methods. - Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797. - https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/ **Scalability Concerns:** The scalability of AGILE with more complex environments is not addressed. It is unclear how well the framework would perform with task complexity.

Questions

**Benchmark Selection:** Why were ProductQA and MedMCQA specifically chosen as the benchmarks for this study? Are there other benchmarks where AGILE could be tested to validate its generalizability? **Human Expert Advice:** Is there a ground truth for when the agent needs to seek advice, and how is this optimized? **Memory Component:** Can you provide more details on how the memory component scales with an increasing number of interactions and how it ensures efficient retrieval of relevant information?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See weakness

Additional questions

I thank you for your reply, that answers some of my questions. I am especially happy that you took the time to perform several seeds of the RL experiments (the results are much more robust now) and that you performed comparison with humans judgements in the answer to Q7 (I now trust the results of evaluations). I also appreciate that you performed additional RL experiments to try and show it helps compared to the strong stf baseline. To make sure I really understand: the original sft policy had 0% [SeekAdvice] rates? And what your new experiment does is start out with higher advice rates and show it leads to more efficient use of human advice (esp when cost of advice falls to 0.1, effectively solving the task by soliciting advice more than half of the time)? As to your remark: > A2: Agents like Voyager are based on prompt engineering I think nothing prevents researchers to build a Voyager agent (with an open weights model) and perform RL on it right? I also appreciate that you emphasize that you build a new task, but from what I can see it doesn't seem very hard since agents (with quite standard architecture, again I think the type of agent you implement is quite common, see also https://arxiv.org/abs/2304.03442) based on 13b models, without RL or search, are able to solve most of it. That's why I was calling for evaluations on other datasets, to be able to compare. (ReAct is not a very strong baseline). I appreciate your new experiments on HotpotQA, and would be interested to know how it compares to the SotA listed in the benchmark homepage (in terms of metrics), and how you implemented advice in the case of this dataset.

Response to authors

Thank you to the authors for the additional results. This response addressed my concerns.