Summary
This work discusses grounding LLM's common sense reasoning capability in physical environments. Specifically, it explores methods to not only generate appropriate actions but also effectively increase interaction efficiency when using Multi-Agent LLM agents in embodied environments. Based on the theoretical background of multi-agent systems, the authors utilize a learned advantage function as immediate feedback for LLM agents. The LLM is prompted to generate actions with high advantage values, and if the generated action plan has low advantage values, it is designed to perform replanning autonomously. The superiority of this framework is demonstrated through experiments across multiple environments.
Strengths
1. Grounding LLM's reasoning prowess in physical environments is a highly promising and crucial field that can help leverage continuously evolving LLMs for real-world problem solving.
2. Addressing interaction costs in embodied environments is a challenging yet practical problem that can resolve various factors, including safety issues and LLM inference costs.
3. The paper is well-organized and written clearly without confusion, making it easily accessible to readers.
Weaknesses
1. The main concern is the novelty of the contribution. Applying MARL's theoretical aspects to embodied multi-agent collaboration appears quite straightforward, and seems independent from the authors' stated goal (as written in Line 77) of enhancing LLMs' reasoning capabilities.
2. While the introduction addresses interaction efficiency as a major issue in prior work, the data collection efficiency of the proposed learning method should also be discussed. This TRPO-style on-policy learning approach is generally known to be less data-efficient than off-policy RL algorithms (even in single-agent settings).
3. I am concerned if the comparison between baselines is fair. For instance, was the same amount of environmental interaction required to create ReAd's training dataset also provided to ReAct and Reflexion?
4. If data collection through interaction is possible, I think the current environmental tasks are relatively simple. For example, using sparse rewards (success/fail) attached to the end of collected dataset trajectories for retrieval-augmented task planning could enhance LLM reasoning more simply and efficiently than the proposed method. Please refer to paper [1]. Could the authors include [1] as a baseline in their experimental comparisons?
[1] LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, CVPR 2023
Questions
1. In Line 38, I wonder about the author's mentions of "lack of task-specific knowledge" instead of domain-specific or environment-specific knowledge. LLMs possess commonsense actionable knowledge (i.e., knowledge to perform tasks) but have never learned the characteristics of the actual environment where this knowledge should be applied.
2. What is the size of dataset \mathcal{D}?
3. Rather than directly fine-tuning the LLM, it uses prompting to employ the LLM as an advantage function optimizer. How is the policy iteration implemented where (i) the LLM (behavior policy) is fine-tuned using the advantage function learned from dataset \mathcal{D}, and (ii) this fine-tuned policy becomes the behavior policy to collect new data? Does dataset collection by the behavior policy occur only once?
4. What is the neural network architecture of the advantage function?
5. Are environment observations converted to text before being passed to the LLM agent?
6. Does using the Advantage function (or Q-function) directly as a policy significantly decrease performance? I'm curious about how a conventional discrete action space RL approach would perform, where Q-values are calculated for all actions and the action with the highest Q-value is selected.