EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

Summary

The paper proposed EnvGen, and LLM based small RL agent training environment dataset generation framework. The method prompts an LLM with game environment task descriptions and objectives, and obtains a more diverse gaming environment configurations for small RL model training. The method also evaluates the performance of the models and provides feedback to prompt LLM to generate game configurations that are more tailored to the RL models in order to improve their performance on the weaker tasks. The paper conducted experiments on two gaming environment: Crafter and Heist, and compared to both LLM based zero shot agents and trained RL based agent. The paper demonstrated that the proposed method only requires a few LLM API calls and can improve RL agents' performance.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. The paper is well written. It clearly explained the proposed method, conducted thorough experiments and demonstrated the benefits of the method 2. Instead of applying LLMs for a specific task, the proposed method used LLM to generate data for more efficient smaller RL model training 3. The feedback loop allows the method to generate customized curriculum for the learning agent.

Reasons to reject

1. To use LLMs to generate training data, it is under the assumption that these generated data is appropriate and close to the real environment. There wasn't any evaluation on the quality of the generated data 2. To demonstrate the benefits of the feedback loop and curriculum learning, it would be more convincing to have some more comprehensive quantitative results other than one example (w/"collect coal", "make stone pickaxe" ) in Figure 6.

Summary

This paper proposes a new framework, EnvGen, that uses large language models (LLMs) to generate training environments for embodied agents. Instead of (1) directly employing LLMs as embodied agents, or (2) directly training RL agents in the environment, the authors propose to use LLM reasoning capabilities to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at. The framework consists of 4 main steps: 1. Ues LLM to generate training environments that allow agents to quickly learn different tasks in parallel. 2. Train small RL agents in the generated environments and potentially learn different skills. 3. Train and measure agent performance in the original environment 4. Use the agent's performance to provide feedback to the LLM, and ask LLm to adapt the generated environments to improve the agent's performance. The experiment results show that with this EnvGen framework, a small RL agent can outperform large LM and RL agents and learn long-horizon tasks significantly faster. **My general review on the paper:** I think this paper is presenting a new perspective on **utlizing** LLMs in embodied learning. Instead of training the LLM or calling LLMs at each step, the authors propose to use LLMs as part of a curriculum learning framework (even though the paper does not explicitly mention this). The LLM is essentially used to generate training environments that help the RL agent learn faster and design curriculum that is tailored to the agent's weaknesses. The experiments demosntrate the effectiveness of the proposed framework, and I think they are comprehensive as it compares with non-LLM and LLM-based methods. Analsysis also show insights on how the method can help tha agent learn more efficiently. There are a few downsides of the paper too. First or all, I think the paper is very closely related to **curriculum learning**, but the authors do not mention this in the paper, nor do they compare with existing curriculum learning methods. Second, the paper does not demosntrate the **upper bound of performance** of this method (e.g., using a larger RL agent or train with much more steps). Currently, the paper only emphasizes the efficiency of the method, but it is not clear how well the method can scale to more complex tasks and larger agents.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The paper presents a novel framework that uses LLMs to generate training environments for embodied agents. 2. The experiments show that the proposed framework can outperform existing methods and help agents learn faster. 3. This is a new perspective on using LLMs in embodied learning, and the paper is well-written and easy to follow.

Reasons to reject

1. A lack of comparison with existing curriculum learning methods (both fixed curriculum and adaptive curriculum). 2. Not enough comparison with other content generation methods. A lot of work are menthioned in the related work section "Deep learning-based game/simulator content generation", but it seems none of them are compared with the proposed method.

Questions to authors

1. Why can the LLM generate training environments that help the agent learn faster? What is the intuition behind this? Is it because the LLM has prior knowledge about the environment or is the LLM only inferrencing from the task description? 2. In page 2, you have "Step 2: We train the RL agent ...", but then "Step 3: We first train the RL agent". The word "first" seems a little confusing since you first trained in the LLm-generated environment in step 2. 3. In Figure 4, you compare (1) Baseline: trained in Crafter for **1.96M** steps, and (2) Ours: trained in **0.96M** steps in Crafter-EnvGen and **1M** in Crafter. Why is this comparison useful? Is training in Crafter-EnvGen more efficient than training in the original Crafter environment? 4. In Section 5, "To help agents generalize to unseen tasks in a text-based dialogue game, Ammanabrolu et al. (2022) augment new tasks with LMs and use a manually designed, fixed curriculum." Do you think the proposed method can be applied to this setting? If so, how would your method compare with the existing method?

Summary

The paper EnvGen, which uses LLM to construct custom training environment configurations for RL agent training in a iterative process. Experiments on Crafter and Heist game environments shows this method allows training RL agent more efficiently and achieve higher performance, while also outperforming baselines which directly promps a LM.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The method provides an novel, interesting perspective in using LMs for environment generation to assist RL agent training, and shows good empirical results. It can be a helpful addition to prior approaches like using LM as reward, using LM as agent, ...

Reasons to reject

- It is unclear if such methods can be applied in broader, more complex scenarios. As noted by the author, “EnvGen also requires that the environment simulators can [skipped] accept configurations in standard formats (e.g., JSON, CSV, YAML, TOML, etc.),". This is in fact a very strong assumption. Games like Minecraft, modern video games like GTA, or environments in other domains like robot training, might have too many degrees of freedom where such high-level configurations do not exist or require extensive human engineering to make.

Questions to authors

- Minor: Figure 5 is a bit confusing to me. While the blue curve is supposed to show how the PPO agent trained on Crafter improves over its 1.96M steps, the x-axis only has 1M steps. It would be good if you can elaborate that in the caption