Empowering Large Language Model Agents through Action Learning

Summary

This research paper introduces an innovative framework named LearnAct, designed to enhance the capabilities of LLM agents through action learning. The study tackles the limitations of fixed action spaces in LLM agents by allowing these agents to learn and refine their actions based on trial and error, a process akin to human experiential learning. This is achieved through an iterative learning strategy where actions are continuously updated based on the outcomes of previous tasks. The authors demonstrate the effectiveness of this approach in the Robotic Planning and Alfworld environments, showing significant improvements in task performance.

Rating

6

Confidence

4

Ethics flag

2

Reasons to accept

1. **Innovative Approach:** The concept of open-action learning to dynamically expand and refine actions for LLM agents is novel and addresses a critical limitation in the current use of LLM agents in various applications. 2. **Thorough Experimentation:** The authors provide extensive experimental results to validate their claims. The use of multiple environments for testing and the iterative approach to learning and refining actions are well-executed and robustly documented. Although the baseline selection is not sound enough. 3. **Potential for Broader Implications:** The study’s findings could have significant implications for the development of more autonomous and efficient LLM agents across a wide range of applications, potentially leading to broader advancements in artificial intelligence.

Reasons to reject

1. **Lack of Comparative Analysis:** While the paper shows that LearnAct outperforms certain existing methods, there is a lack of comprehensive comparative analysis with a wider range of existing solutions (especially SOTA and more up-to-date methods), which could provide a better understanding of LearnAct’s positioning in the field. 2. **Complexity and Replicability:** The complexity of the LearnAct framework and its dependency on specific environments and configurations may limit its replicability and broader application. The paper could benefit from a clearer explanation of how LearnAct can be adapted or generalized for other contexts. 3. **Potential Overfitting:** There is a concern about the potential for overfitting in the iterative learning process, especially given the significant improvements reported. The paper could be strengthened by including a discussion on measures taken to avoid overfitting and ensure the generalizability of the learned actions.

Questions to authors

1. **Comparative Analysis:** Could the authors provide a more detailed comparative analysis with other methods in action learning for LLM agents? Specifically, how does LearnAct perform compared to methods that do not use iterative learning strategies? 2. **Baseline Selection:** The authors claimed that LearnAct achieves more effective and efficient learning compared to state-of-the-art methods, but the baselines are not state-of-the-art and up-to-date enough. In fact LearnAct performs worse than some up-to-date models, such as FireAct [6] and KnowAgent [7]. 3. **Flexibility:** Is it necessary to “Expanding Action Space by Action Creation” first? If not, how can you adaptively decide to expand the action space or not? If so, could you explain why action space expansion is a necessity, as the current action space is sufficient enough in many cases. 4. **Generalizability:** Can the authors elaborate on how the LearnAct framework can be adapted for use in environments or tasks significantly different from those tested (e.g., non-robotic applications)? 5. **Scalability Concerns:** How does the LearnAct framework scale with increasingly complex tasks or larger environments? Are there computational constraints that could limit its practical applicability? 6. **Overfitting Measures:** What measures have been implemented in the LearnAct framework to prevent overfitting during the iterative learning process? How do you ensure that the actions learned are not overly tailored to the specific training examples used?

Ethics concerns details

The newly-generated actions may require ethics check

Summary

The authors present alba approach for LLM agents to learn new actions and thereby improve the ability to successfully perform tasks in a simulation. The presented idea to learn new actions formalized as python code and how to integrate this new ablities into the policy of the agent. The approach is overall presented clearly, however some details could be explained in more detail. The approach is very well evaluated on two different tasks (Robotic Planning and AlFWorld). Thereby, the authors compare the method to several related work and the presented results are promising. The authors missed a detailed analyze of the influence of the LLM. The LLM is only treated as a black box. The influence of the quality of the LLM is only analyzed very briefly and the authors do not evaluate any open source LLM to perform a more detailed analyze

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

Good motivated approach to improve the performance of LLM Agents Detailed analysis of the presented approach Promising results

Reasons to reject

The influence of the LLM on the overall performance is only investigated very briefly.

Questions to authors

Tabel 5: How can the model use learn actions before learning?

Summary

This paper is about improving the performance of language agents by refining the action space during the learning phase. Specifically, the authors propose LearnAct, an iterative system that uses LLMs to create new (or edit existing) actions based on past experiences and to act as a policy. In this work, the actions are expressed as Python code which enables the use of conditional statements, loops, and function calls (i.e., composition of actions). Experiments were conducted on robotic planning tasks and ALFWorld (a text-based game version of ALFRED, a 3D embodied environment). Empirically, the authors show that LearnAct achieves better performance on average compared to some relevant baselines (ReAct, Reflexion, CodeAsPolicy, Voyager) on those environments. Edit: I have increased my score to 7.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

- Letting the the action space evolve based on past experiences is a nice idea. This is a good way to get actions that operated at different level of abstraction, i.e. hierarchical planning. - Having higher-level actions (expressed as code) help reducing the compute cost during inference. That is, you don't have to call the LLMs at each timestep. - Transferability of the learned actions to new tasks from a same domain. I think this is a very important aspect of the work, as many prior works focuses on single-problem solving.

Reasons to reject

- The research on Language Agents is evolving very fast and there is probably several prior works missing from the related work section. On top of my head, two came in mind: [TaskWeaver: A Code-First Agent Framework](https://arxiv.org/abs/2311.17541) and [ReGAL: Refactoring Programs to Discover Generalizable Abstractions](https://arxiv.org/abs/2401.16467). I think it would be beneficial to the reader to have a more comprehensive related work section to better position their work. - This is a smaller concern. I noticed a small paragraph in A.6.3 about failure mode of the proposed technique regarding excessive iterations (which according to Fig.3 anything above 2 iterations looks worst!). Is this that the only failure? I doubt it, e.g., limit of the context length in ActionLearn to update the full action space at each iteration. I think having a proper discussion of the limitation as part of the main text would strengthen the paper.

Questions to authors

- What is the definition of closed-loop in the context of this work? Is it the action learning process that is closed-loop or the actions (i.e., Python functions) themselves? If you are referring to the actions, have you checked if they are really closed-loop? In my experience Voyager tends to generate actions that are open-loop, i.e., they ignore the state of the environment as they send command to the environment. Granted this might be less problematic in your environments compared to Minecraft where other entities are moving in parallel. - In section 5.3, it says the **temperature** is 0, but it also mentions sampling number is 4. I'd expect to get very similar samples (if not all the same). Can the authors clarify this? - Do the authors have any insights on the generalization of the learned actions to unseen domains but that are sharing the same predefined actions? - I'm curious to know if the authors have considered any open-source models during their experimentation?

Summary

This paper proposes a training framework that instructs LLMs to learn new actions. During the training stage, the action is created in the form of executable code function, and is refined through execution feedback. In the inference time, the LLM can utilize the created action to tackle more novel scenarios and perform more complex tasks.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

1. The idea is interesting, and can potentially solve the problem of constrained action space. 2. The experimental results demonstrate the effectiveness of the proposed framework.

Reasons to reject

Overall, the idea is intuitive and makes sense to me. But the biggest concern is that I do not find any fundamental difference and novel parts that distinguish this work from some previous publications, including but not limited to CRAFT, and Voyager. Please correct me if I have any misunderstanding. Basically, I think the fundamental part is creating some tools or actions defined in this work during the training time that are reusable, which can help LLMs to address some special test cases. Which part is novel in this work?

Questions to authors

None

Discussion period

Hi reviewers! This is a friendly reminder that the authors have posted responses to the initial set of reviews. I'd appreciate it if you could look through the authors' comments soon and let us know if your concerns have been addressed / if there are any outstanding issues soon so there's time to discuss. With appreciation, Your AC