Summary
The paper introduces RL-GPT, a novel framework that integrates Large Language Models (LLMs) with Reinforcement Learning (RL) to enhance the performance of LLM-based agents in complex, embodied environments. The primary goal is to address the limitations of LLMs in executing intricate logic and precise control, especially in open-world tasks like those found in the Minecraft game.
The RL-GPT framework employs a two-level hierarchical approach, consisting of a slow agent and a fast agent. The slow agent is responsible for decomposing tasks and determining which actions can be coded directly, while the fast agent generates the corresponding code and integrates RL configurations. This division of labor allows for efficient handling of both high-level planning and low-level execution.
Key contributions of the paper include:
1. Two-level hierarchical framework to determine which actions should be learned by RL and which ones can be coded (e.g. in a python).
2. The introduction of a mechanism where high-level actions coded by LLMs are appended to the RL agent’s action space, instead of only relying on low level actions.
3. Empirical validation showing that RL-GPT outperforms traditional RL methods and existing LLM agents in the Minecraft environment, particularly excelling in the ObtainDiamond task and other MineDojo tasks.
In experiments, RL-GPT demonstrated superior performance, achieving state-of-the-art (SOTA) results in several tasks, including rapidly obtaining diamonds and excelling in long-horizon tasks with limited computational resources.
Strengths
**Originality**:
- The paper proposes a novel integration of RL and LLMs, leveraging the strengths of both to improve task efficiency in open-world environments.
- The fast agent's method of combining code-as-policy with RL by inserting high-level coded actions into the RL action space is a creative and original solution that sets RL-GPT apart from other frameworks.
**Quality**:
- Claims are well supported by extensive experimental results, comparing with previous SOTA methods in MineDojo
**Clarity**:
- The paper is generally well-written and structured, with detailed explanations of the framework components and their interactions.
- Figures and tables effectively illustrate the performance improvements and workflow of RL-GPT.
**Significance**:
- There is often a question about the what is the right abstraction level for LLM agents to operate on: high level APIs/code-as-policies, or low level mouse/keyboard, etc. RL-GPT addresses this by trying to leverage the both, with the higher level actions appended to the RL agent’s action space.
- The state-of-the-art performance in MineDojo tasks highlights its potential impact on the field, though the focus on Minecraft limits the generalizability of the results.
Weaknesses
1. The work focus on Minecraft, while useful as a testbed, may not fully represent the range of challenges present in other open-world environments. Expanding the scope of testing to other domains would enhance the significance of the work
2. While the integration approach is novel, the individual components (LLMs, RL, task decomposition) are well-known techniques. The paper could benefit from a more explicit discussion of how these components are synergistically combined to create a unique solution.
3. Some sections, particularly those describing the two-loop iteration process and the role of the critic agent, could benefit from more detailed explanations or examples to improve understanding. I am aware that the full prompts are included in the appendix and they do not fit in the main paper, however.
Questions
1. The slow agent can all a sub-action that the RL agent executes. But can the fast agent which uses “code as policies” also call the RL agent inside its loop?
2. What is the architecture and inputs of the PPO RL agent? In particular, it was said that “PPO is limited to a set of skills”: how is the sub-action (e.g. “Harvest log” sub-action 2 in Figure 3) from the slow agent represented to the policy network? Is it just a one-hot vector (“PPO is constrained to a limited set of skills”), or are you encoding the sub-action text instruction as an embedding vector, etc., or are you learning a different set of policy weights for each sub-action, or something else?
3. How would the proposed framework adapt to other open-ended environments beyond Minecraft? Are there specific modifications required for different domains?
4. (RL Interface Ablation) Do you have any hypotheses on why the action space reconstruction is more effective than designing the reward function? Is the action space shortening the horizon of the task and makes the reward less sparse?
5. Table 4: is this showing the %? In Section 4.4, it was reported as 0.34% and 0.65% success rate. Was this a typo and it’s supposed to be 34% and 65% success percentage for crafting a table?
Post rebuttal: I have increased the score after the rebuttal.
Limitations
The authors acknowledge some limitations of their work, notably the reliance on the capabilities of LLMs and the potential challenges in generalizing to unseen data. However, there are additional limitations and assumptions that could be more explicitly addressed:
1. The approach assumes an environment with a structured observation space that can support coding as policies. This might not be feasible in more complex or less structured environments. Environments need to be compatible with the two-level hierarchical framework and provide sufficient information for task decomposition and action coding. It’ll be helpful to provide guidelines for adapting the framework to different types of environments, including those with less structured observation spaces.
2. The framework's reliance on multiple interacting agents and iterative refinement could complicate implementation and debugging. The need for specific prompts and hand written examples add to the complexity, making it challenging for broader adoption.