Building Cooperative Embodied Agents Modularly with Large Language Models

Summary

This work proposes a framework called Cooperative Embodied Language Agent (CoELA), which explores the potential of Large Language Models (LLMs) for multi-agent communication. The framework is composed of five modules, among which the communication module helps the agent cooperate more effectively than traditional methods. Experiments conducted on TDW-MAT and C-WAH have demonstrated the effectiveness of CoELA when compared to strong planning-based methods, showcasing effective communication. Furthermore, the authors also explore the potential of using open LMs as LLMs, which is an impressive aspect of their work.

Strengths

1. The comprehensive experiments demonstrate that the proposed method can efficiently cooperate with other agents, which is very impressive. 2. The presentation is clear.

Weaknesses

1. Details regarding how CoELA cooperates with other agents are lacking. Sections 4 and 5 do not mention the mechanisms for cooperation between agents, such as how one MHP cooperates with another MHP or with CoELA. 2. Section 5.3.1 is missing the results of CoELA when driven by CoLLAMA. 3. The authors appear to be focused on exploring the potential of using Large Language Models (LLMs) in the Discrete Execution setting through communication. However, it's important to consider that communication may lead to challenges in certain scenarios, such as agents failing to reach a consensus. A discussion of such scenarios seems to be missing.

Questions

1. How does the traditional MHP cooperate with MHP or CoELA? 2. How does CoELA handle the scenario where no consensus is reached? For example, Alice wants Bob to goto A and Bob wants Alice to goto B.

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper proposes a modular framework that integrates the Large Language Models to build Cooperative Embodied Language Agents CoELA, which focuses on the multi-agent setting with decentralized control, complex partial observation, costly communication and multi-objective tasks. Empirical experiments on C-WAH and TDW-MAT show that CoELA can achieve promising cooperative performance. Additional experiments with real humans demonstrate that CoELA can earn more trust and cooperate more effectively with humans.

Strengths

1. This work proposes one feasible approach to building embodied agents with Large Language Models which seems to be sound. This demonstrates the prosiming potential of LLMs to build cooperative embodied agents and inspires future research well. 2. The experimental setup is relatively comprehensive, including cooperative evaluation with AI agents and real humans. Besides, the discussion of the experimental results is also very interesting and thorough. 3. The discussions about failure cases and limitations are appreciated.

Weaknesses

1. Despite the fact that the structure of the manuscript is organized well, the description of the method is relatively brief, with some details not sufficiently elaborated. For example, the manuscript states that, for the execution module, CoELA will utilize the procedure stored in its Memory Module to execute the high-level plan. However, it is unclear what form these procedures take and how they are obtained for a specific environment. 2. If I'm not mistaken, given a specific environment, CoELA needs to manually design/list the possible high-level plans, which may be a relatively tedious workload. Similar issues may arise when determining the possible high-level plans on each state and defining the procedure for each plan. This significantly influences the generality of the method and it may be challenging to implement these manual work in complex problems. Besides, listing the possible plans may significantly increase the prompt length, which might influence the performance. 3. The baselines in the experiments are relatively simple and heuristic. More baselines, especially methods designed for embodied agents, are recommended.

Questions

1. How does CoELA determine the valid high-level plans at each state? Is it determined manually? 2. Have there been previous works using LLM for implementing embodied agents? Can more discussion or even experimental comparisons be made with these works? What are the core contributions and innovations of CoELA compared to them?

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Ethics concerns

N/A

Summary

This paper presents an innovative approach to address multi-agent cooperation challenges in decentralized settings with costly communication and raw sensory observations. The authors introduce a modular framework integrating Large Language Models (LLMs), resulting in the Cooperative Embodied Language Agent (CoELA), capable of efficient planning, communication, and cooperation.

Strengths

1. The paper introduces a unique integration of Large Language Models within a modular framework for decentralized multi-agent cooperation, addressing practical challenges in varied embodied environments. 2. Robust empirical support is provided through comprehensive experiments and a user study, showcasing the effectiveness of the approach and its positive impact on human-agent cooperation. 3. The paper is well-articulated and structured, offering clear insights and setting a strong foundation for future research in multi-agent cooperation with embodied agents.

Weaknesses

1. This method assumes a skill library that is manually defined for a specific domain, i.e. execution module. However, this limits its applicability in other domains where predefined skill libraries are not available. 2. The pipeline appears to be quite complex and relies on several hand-defined modules, including perception modules, three memory modules, planning, and execution. Are all of these modules necessary? Conducting an ablation study would provide better understanding. 3. The experimental design lacks breadth as it only considers one scenario. Including evaluations across multiple scenarios would strengthen the findings. 4. There is some ambiguity regarding the execution module, memory module, and perception module details. Were they all designed by humans using language-based approaches?

Questions

See in weakness.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Summary

The manuscript presents a comprehensive study on constructing cooperative embodied agents using Large Language Models (LLMs), aiming to address multi-agent collaboration in decentralized settings with challenges like raw sensory observations, costly communication, and multi-objective tasks. The authors introduce CoELA, a Cooperative Embodied Language Agent, which integrates the LLMs' capabilities with a modular framework encompassing perception, memory, execution, communication, and planning. The system's performance is evaluated in two embodied environments: C-WAH and TDW-MAT, demonstrating its ability to outperform planning-based methods, particularly when driven by GPT-4. The use of natural language for agent communication is highlighted as a significant advantage, fostering trust and effectiveness in human-agent interactions.

Strengths

**Robust Motivation:** The paper is grounded in a strong and compelling motivation to enhance agent collaboration in complex environments. By addressing the need for agents to effectively communicate and plan their actions in a coordinated manner, the authors establish a solid foundation for their research, showcasing a clear understanding of the challenges and opportunities in the field. **Carefully Designed Pipeline:** The system architecture demonstrates a thoughtful and meticulous design, integrating multiple modules and dual language models to manage both communication and planning. This comprehensive approach ensures that each aspect of the agent's interaction is given due consideration, resulting in a pipeline that is both balanced and well-reasoned. The deliberate inclusion of separate models for different functions reflects the authors' dedication to creating a system that is tailored to meet the specific demands of agent collaboration. **Thorough Analysis and Discussion:** The paper excels in providing an in-depth analysis and discussion of the results, helping readers to fully grasp the implications and nuances of the study. The authors do not shy away from addressing the limitations of their work, offering a balanced view that adds credibility to their findings. This level of detail ensures that the paper serves not only as a presentation of the proposed framework but also as a valuable resource for future research, encouraging further investigation and innovation in the field of agent collaboration.

Weaknesses

**Complex Model and System Design:** The architecture of the system is intricate, requiring each agent to manage five different modules and two distinct LLMs for handling communication and planning. This complexity can lead to instability in the LLM's performance, especially when processing lengthy textual inputs describing complicated scenarios. Furthermore, the challenge to maintain scalability becomes apparent as the addition of objects and details can potentially overwhelm the system, limiting its extendability. The intricate design also indirectly contributes to the limited utilization of spatial information and the difficulty in effective reasoning over low-level actions, as these scenarios would require even longer prompts. **Effectiveness of Communication:** The effectiveness of communication within the system seems to be suboptimal according to the ablation study. Therefore, a natural question is whether there is a possible improvement for the communication module. Personally, I am interested in the issues in Figure 5a. I am curious if Alice misinterprets Bob's actions due to an incorrect perception or what. Will bi-directed communication, such as Alice asking Bob if he has placed the object in the container when she needs to know, plus Bob telling Alice what he is doing at the beginning and the end of one mission, might serve as a straightforward solution to the problems, highlighting the need for a more responsive and interactive communication system. By addressing these weaknesses and optimizing the system accordingly, there is potential to enhance the performance and scalability of the framework, leading to more effective agent collaboration.

Questions

1. How to evaluate the communication cost? Is there any tradeoff study on communication cost and effectiveness? 2. How does the system perform in scenarios with an increased number of objects and more complex interactions, and what measures are in place to maintain scalability? 3. For turning left/right, what will happen if the object in interest is on the back of the agent? How is the visual-related textual input organized? Did the work try to use the oracle information of the whole environment, including objects and relations?

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

General Response to All Reviewers and ACs

We thank all the reviewers and ACs for their time and efforts in reviewing our paper and giving insightful comments. In addition to the detailed response to specific reviewers, here we would like to highlight our contributions and clarify some concerns. **1.Our Contributions** We are glad to find that the reviewers have acknowledged our following contributions: - **Promising idea of incorporating LLMs into modular frameworks to enhance agent collaboration in challenging environments** - Carefully Designed Pipeline that is both balanced and well-reasoned [tquX] - a unique integration of Large Language Models within a modular framework for decentralized multi-agent cooperation, addressing practical challenges [2bPF] - demonstrates the promising potential of LLMs to build cooperative embodied agents [iNRX] - the proposed method can efficiently cooperate with other agents, which is very impressive [xiUJ] - **Comprehensive experiments design and in-depth analysis** - The paper excels in providing an in-depth analysis and discussion of the results [tquX] - Robust empirical support is provided through comprehensive experiments and a user study [2bPF] - the discussion of the experimental results is also very interesting and thorough [iNRX] - **Valuable insights for future research** - a valuable resource for future research, encouraging further investigation and innovation in the field of agent collaboration [tquX] - offering clear insights and setting a strong foundation for future research in multi-agent cooperation with embodied agents [2bPF] - inspires future research well [iNRX] - the authors also explore the potential of using open LMs as LLMs, which is an impressive aspect of their work. [xiUJ] **2.Clarifications** - **The complex design** - The framework is **deliberately** designed to incorporate LLMs into **modular** frameworks mimicking human cognitive architecture to deal with challenging decentralized multi-agent cooperation scenarios. - Experiments on two challenging embodied AI benchmarks show our modular framework is **effective** and can be **adapted** to different observation and action spaces. - The ablation study and in-depth analysis in the paper show **all modules contribute to the best performance** of CoELA when cooperating with other agents and humans. - **Communication Efficiency** - Communication in our setting takes time as in real life, so the agents need to **reason over the pros and cons of each step of communication on its own**. - Our designed framework can show **effective communication behaviors** under the challenging setting, **especially when cooperating with humans**. - This setting also leads to many interesting scenarios, such as where agents fail to reach a consensus as reviewer xiUJ suggested. We analyze the agent's behaviors exhibited in the experiments and show CoELA are **prone to cooperate** and coordinate plans without arguing back and forth which can be credited to LLMs trained to follow instructions and trust their cooperators. - **The scalability of the framework** - Our evaluated scenarios are **complex** as there are **18 types of objects and containers of interest** in the **multi-room scenes**, and **6-7 actions to interact with**. - Across our experiments, the prompt uses **453** tokens on average and **591** tokens at max, which is rather **economical**, thanks to our deliberately designed memory module to store and organize the objects of interest **effectively**. - Considering even more complex scenarios with too many objects of interest, our framework can easily integrate some more deliberately designed filter modules for better scalability, thanks to the **modular design** of our system. - **Predefined skill library** - Our low-level skills are obtained from motion planning, which is common in embodied AI benchmarks, given the focus is more on high-level reasoning and planning. - Our framework is **flexible** thanks to our **modular design**, and can be **easily adapted** to various low-level control policies, including reinforcement learning and trajectory optimization, offering a distinct advantage over end-to-end methodologies. We hope our detailed responses below convincingly address all reviewers’ questions. **Revision Summary** - We add an additional discussion paragraph in Appendix D to discuss in depth about the related work of using LLMs for embodied planning as Reviewer **iNRX** suggested. - We add more details about the cooperation mechanisms in Appendix C.2 as Reviewer **xiUJ** suggested. - We add an additional discussion paragraph about CoELA's communication behavior in Appendix D as Reviewer **xiUJ** suggested.

Thank you and we are looking forward to your post-rebuttal feedback!

Dear AC and all reviewers: Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity. The discussion phase has been on for several days and we are still waiting for the post-rebuttal responses. We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice. Best, Author