Building a general-purpose agent is a long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through a variety of long-horizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises historical information into Abstracted Multimodal Experience Pool that provide agents with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, a multimodal agent, Optimus-1, is constructed with dedicated Knowledge-guided Planner and Experience-Driven Reflector, contributing to a better planning and reflection in the face of long-horizon tasks in Minecraft. Extensive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging long-horizon task benchmarks, and exhibits near human-level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hybrid Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
Paper
References (63)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This paper introduces Optimus-1, a retrieval augmented generation method that enables minecraft agents to excel in long-horizon tasks. The proposed method is based on a Hybrid Multimodal Memory Module that consists of an Abstracted Multimodal Experience Pool and a Hierarchical Directed Knowledge Graph. Those two key components are mainly proposed to solve the challenge of insufficient structured knowledge and lack of multimodal experience. Empirical evaluations show that such a memory module can effectively improve the performance of a minecraft agent in long-horizon planning tasks. Ablative experiments are conducted to validate the effective of each component.
Strengths
This paper addresses an important problem that transforms LLM into autonomous decision making agents. The proposed method achieves a significant improvement over prior state-of-the-art in Minecraft, particularly exceling at long-horizon tasks such as getting diamonds.
Weaknesses
The writing of this paper can be improved. In particular, I would suggest a major rewriting of the method section that 1) follows a top-down organization to first talk about high-level ideas of the method before going to the details such as "1 frame per second" and "a window size of 16", and 2) reduces unnecessary complications of the terminology including Hybrid Multimodal Memory Module, Hierarchical Directed Knowledge Graph, etc. Major claims in the introduction are not supported by empirical evidence. It is unclear whether GPT4V etc does not have sufficient structured knowledge and related works in multi-modal agents do employ multi-modal experiences [1]. It seems that the use of Hierarchical Directed Knowledge Graph is limited to Minecraft where there is a strict Directed Graph relation between different objects. It is unclear whether this method can be helpful in the general setting. The performances are evaluated on a set of custom benchmarks, and it will be good to have results on prior benchmarks reported by the baseline methods such as Voyager and Jarvis. Minor: Line 25 "Early research [1, 6, 17] developed simple agents by constructing policy networks." seems unclear what those policy networks are. Line 30, what long horizon tasks are the authors talking about? Line 36 grammar error "Insufficient of Structured Knowledge", what is empirical evidence for existing agents do not have structured knowledge? Line 70 and line 71, two 30% seems repetitive. Line 108, where is the subgoal coming from? Line 115-116, what are those reflection phases? In line 206, why did the authors contruct a new benchmark instead of following the prior benchmarks on minecraft? [1] Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., … Yu, G. (2023). AppAgent: Multimodal Agents as Smartphone Users. [2] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023 [3] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997, 2023.
Questions
See weakness
Rating
6
Confidence
4
Soundness
1
Presentation
1
Contribution
2
Limitations
Limitations are not adequately discussed in the paper. One potential limitation is that the performance of RAG methods may be capped by the performance of the base model.
Responses to Reviewer wzp9 (2)
This comment connects to the Responses to Reviewer wzp9 (1) > Q5: It is unclear whether such designs can generalize to more realistic settings like real-world web navigation agents or robotics. A5: (1) To address your concerns about the generalisation of our method in real-world scenarios, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [2] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents. Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks. | Method | Success Rate | | --- | --- | | GPT-4 | 48.9% | | AppAgent | 73.3% | | Optimus-1 | **86.7%** | (2) Proposed Hybrid Multimodal Memory is a general architecture. As mentioned in lines 31 to 35 in the manuscript, we were inspired by the theory that 'humans benefit from knowledge and experience when performing long-sequence tasks' , and propose a novel Hybrid Multimodal Memory structure that incorporates both knowledge and experience into the memory mechanism. We argue that it is general and can be adapted according to different environments. In the experiments mentioned above in the app scenario, the key step is obtaining the logical relationships between buttons or actions and converting them into a knowledge graph. Once these logical relationships are established, HDKG can easily be adapted to the app environment. As for AMEP, it can be simplified to store the task prompt, images, and actions for each atomic operation. We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. If you have any further questions, please feel free to contact us. [1] Wang et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. 2023. [2] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.
Thank the authors for the additional response on the clarity of the paper and results on real-world app navigation. I am more convinced of the proposed method being general and increase the score. I hope that those new results and rewriting can be reflected in a future version of the paper.
Responses to Reviewer wzp9
We are pleased to have addressed your concerns. We will incorporate your valuable suggestions in future revisions and add more experiments in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).
Summary
This paper presents Optimus-1, a multimodal agent that focuses on Minecraft tasks. Specifically, Optimus-1 is equipped with a Hybrid Multimodal Memory including: 1) a Hierarchical Directed Knowledge Graph that stores the world knowledge through free exploration and teacher guidance; 2) an Abstracted Multimodal Experience Pool that enables Optimus-1 to reason about the current situation by using past experience. Based on the multimodal memory, Optimus-1 adopts a Knowledge-Guided Planner and an Experience-Driven Reflector to generate better plan and reflect periodically in long-horizion tasks. Experiments results illustrates the effectiveness of the multimodal memory of Optimus-1 in long-horizion tasks.
Strengths
1. The paper introduces a multi-modal memory mechanism that includes a hierarchical world knowledge graph and a multi-modal past experience pool. The memory is later utilized by the multi-modal planner and reflector module of the Optimus-1. 2. Good experiment results illustrate the effectiveness of Optimus-1 compared to other strong baselines. Sufficent ablation studies of the effectiveness of the modules in Optimus-1 as well as the necessity of both success and failure cases for reflection. 3. The paper is well written.
Weaknesses
1. The details of the construction procedure of AMEP are unclear (e.g. How to maintain the image buffer by computing image similarity? What is the threshold used in MineCLIP? ) 2. How Optimus-1 acquires world knowledge through free-exploration is not specified. 3. There are other minecraft agents (Jarvis-1, etc.) using multi-modal memory. The paper claims the efficiency of the memory storage and retrieval compared to Jarvis-1, but this is not quantitatively evaluated. 4. It seems that the low-level action controller will not be updated through reflection, which limits the effectiveness of the reflection pipeline.
Questions
1. In the construction of AMEP, given the video stream, how to adaptively update the abstracted frames in the buffer? How is the threshold of MineCLIP determined? 2. In the free-exploration phase, what is the efficiency of Optimus-1 to learn an entry of world knowledge such as "a stone sword can be crafted with a wooden stick and two cobblestones" through random exploration? 3. In the reflection phase, how to retrieve past experience from AMEP? Is the retrieval based on the image similartiy, task goal, or a combination of both? How does the fail case contribute to the final success of the task? 4. In Table 1, how is GPT-4V evaluated? Does GPT-4V have the same world knowledge as Optimus-1?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes.
Summary
The paper proposes "Optimus-1" a Mulit-Modal LLM based agent. They evaluate their agent extensively on MineCraft and demonstrate superior performance compared to previous work. The key components of the approach are: 1. A memory module consisting of structured memories (DAG) and multi-modal experience replay (with negative and positive examples) 2. A planing module 3. A reflection module 4. A "low-level" execution module The novelty of the work consists of: 1. The DAG memory 2. the positive & negative samples in the replay memory 3. Putting everything together in this way Multiple ablation studies are made.
Strengths
Strengths: 1. A lot of work went into running these experiments and a lot of results are presented (incl. open-source MLLMs) 2. A lot of work went into designing the evaluation benchmark 3. A novel agent that performs better than previous work (and seems that can be applied to more tasks).
Weaknesses
Weakness: 1. "teacher guidance learning" for HDKG -> It seems that teacher demonstrations are needed to make the agent perform the hard tasks. a. The paper (even incl. appendix) does not explain where these demonstrations come from and what the impact of these is. (Or how costly these demonstrations are to obtain). b. Looking at ablation studies one can see that this Knowledge Module is key and performance drops by 20% (however, it is unclear how much the "teacher" contribution is. It seems that without the teacher contribution the model might perform quite poorly compared to baselines.) Therefore this indicates that expert human demonstrations are needed to actually make the agent work, as opposed to the "LLM" doing the reasoning and work. c. Similarly, all further evaluations (incl. against other benchmarks in the appendix) are therefore somewhat questionable. 2. The actual learning mechanism of the DAG memory (HDKG) is not described in sufficient detail to replicate the work. It would be good to cite more references or explain this part in more detail. 3. Positive and Negative examples in AMEP. While the paper argues this is an important contribution, it seems that it is nowhere evaluated if only positive examples would be more succesful. 4. Computational costs are only marginally mentioned ($5000 for OpenAI, and only 4xA100, but no amount of hours). It is not clear how much the "exploration" / learning phase is (time and money), what about the teacher phase? 5. Evaluation outside of Minecraft would be interesting as well.
Questions
Questions: 1. How are teacher demonstrations obtained? How costly is this? How many human annotators and hours are needed for this? 2. How does the agent perform without teacher demonstrations? 3. What is the performance without negative samples in AMEP? 4. How long does the training / exploration phase take? How costly is it? (How many GPU hours on 4x A100 did it take?) How much is an evaluation? 5. How long would your method take to setup in a new environment altogether (and what would be the rough steps)? What is another good environment?
Rating
8
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations: 1. The authors speak about the limitations of one of the modules (the action generator), however, the teacher phase limitation mentioned above is not discussed. It seems that the method strongly depends on expert demonstrations and these are expensive to obtain in new environments outside of Minecraft. 2. Evaluations on other envs than Minecraft would have been interesting to discuss.
Responses to Reviewer mEzT
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows: > Q1: How could you illustrate the automatic plan acquisition step more clearly? What is required beforehand (eg a Wiki, in what format?) what are the exact steps (i.e. input and output of each component). A1: **Step 1: We randomly select 5 tasks for each Group (7 groups in total) that are not included in the benchmark.** **Step 2: For each selected task, we use a script to automatically obtain the crafting relationships from the Minecraft Wiki**. The pseudocode for the script is as follows: ```python def get_information_from_wiki(item): html = get_html_from_wiki(f"https://minecraft.wiki/w/{item}") status, recipe = parser_crafting_from_html(html) if status == "failed": # means atomic item breaking = parser_breaking_from_html(html) can_break_tools = parser_tools_from_breaking(breaking) save_breaking(item, can_break_tools) return save_recipe(item, recipe) for sub_item in recipe: get_information_from_wiki(sub_item) # get item's knowledge from wiki item = "wooden_sword" get_information_from_wiki(item) ``` Taking the task “craft a wooden sword” as an example, we use the script to automatically obtain the crafting relationships: {1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword}, {1 log → 4 planks}, {2 planks → 4 sticks}, {4 planks → 1 crafting table}. **Step 3: These relationships are converted into a directed acyclic graph through the script below**. ```python def get_knowledge_graph(item): status, recipe = read_recipe(item) if stauts == "failed": tools = read_breaking_tools(item) for tool in tools: tool_graph[tool][item] = True return for sub_item in recipe: craft_graph[sub_item][item] = recipe.number[sub_item] get_knowledge_graph(sub_item) # get item's knowledge graph item = "wooden_sword" craft_graph = {} tool_graph = {} get_knowledge_graph(item) # merge 2 graph into an unify knowledge graph kg = merge(item, craft_graph, tool_graph) ``` By performing a topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2) **Step 4: We prompt GPT-4 to construct a plan in order from basic materials to advanced materials**: ```python System Prompt: You are an expert in Minecraft and you can efficiently make plans for me to complete challenging tasks. User Prompt: I need to complete the task {Task}. Here are the materials needed and their quantities {Materials}. Please make a feasible plan for me in the order from basic materials to advanced materials: ``` Finally, we get the plan: {1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4. Craft a wooden stick 5. Craft a wooden sword} We have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). And we will add these implementation details in the Appendix. > Q2: Does Optimus-1 exceed the capabilities of the teacher alone? How meaningful are these new capabilities? A2: Actually, 'teacher' refers to the internal knowledge of the Minecraft environment. In the free exploration phase, Optimus-1 acquires basic knowledge through environmental feedback, while in the teacher guidance phase, Optimus-1 obtains advanced knowledge through the automated process described in A1. This knowledge cannot directly transform into the capabilities in Minecraft. So we constructed Optimus-1, which includes modules such as the Knowledge-guided Planner, Experience-Driven Reflector, and Action Controller, to transform this knowledge into the capabilities for the agent to execute long-horizon tasks in Minecraft, through the reasoning and reflection of multimodal large language model. **Figure 5(b) in the manuscript** shows that Optimus-1's performance continually improves during the multi epoches 'free exploration-teacher guidance' learning process. This indicates that Optimus-1 can utilize past memories (knowledge and experience) to gradually enhance its performance on unseen tasks. **Table 2 in the manuscript** shows that the performance of Optimus-1 significantly decreases after removing such knowledge (e.g., from 9.5% to 1.8% on the Diamond Group). Therefore, knowledge has a significant impact on Optimus-1's capabilities to execute long-horizon tasks. We hope that these explanation can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.
Thank you for this detailed clarification and answer! The score is raised from 7 to 8. We would recommend adding all of the above into the paper to make the paper clearer. Specifically: 1. The comparision against previous work (such as Voyager-1) 2. The detailed explanation of the various steps and how they work and what they involve. 3. Additional discussion above.
Responses to Reviewer mEzT
We deeply appreciate the time and effort you invested in the evaluation of our paper. We will revise the paper based on your and the Reviewer wzp9's suggestions, including: 1. Add the comparison with existing Minecraft agents (e.g., Voyager) in Appendix E.3. 2. Add a detailed explanation (how to construct and apply) of the HDKG and AMEP in Appendix F.1. 3. Add experimental results in the app agent scenario in the Appendix. 4. Revising the methodology section and other parts. 5. Incorporating clarifications and implementation details discussed with the reviewers into the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). Thank you again for your valuable suggestions, which are crucial for improving the quality of our paper.
Summary
The paper tackles the long-horizon tasks in Minecraft by building a pipeline based on multimodal LLM. Specifically, it proposes to store multimodal memory during agent exploration and a knowledge graph that stores the causal relations between objects and tasks. Additionally, a self-reflection mechanism is used to improve the textual actions. Finally, the textual actions are executed by a pre-trained action controller to output the low-level actions.
Strengths
- The core idea is intuitive that multimodal experience can help with long-horizon tasks for embodied agents, where they can be retrieved at test time to help with decision making. - The paper is overall written clearly and sufficient details are provided in appendix.
Weaknesses
- There seem to be too many individual components in the paper, most of which are claimed to be important components in the paper, which makes it unclear to what extent the claimed contribution generalizes to broader settings beyond Minecraft (or even just beyond the evaluated tasks in Minecraft), as it is also highly plausible that the entire pipeline is carefully designed specifically for the evaluated tasks.
Questions
See "weaknesses" section above.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
The limitations are described in the paper.
Response to Reviewer 397y
### Additional experimental results To demonstrate that our Hybrid Multimodal Memory is easily adaptable to other domains, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents. Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks. | Method | Success Rate | | --- | --- | | GPT-4 | 48.9% | | AppAgent | 73.3% | | Optimus-1 | **86.7%** | We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. We welcome the reviewer to continue the discussion with us during the discussion phase. If you have any further questions, please feel free to contact us. P.S.: We just realized that it was not visible to you due to a setting error. We sincerely apologize for any inconvenience this may have caused and for not responding to your question in a timely manner.
Response to Reviewer 397y
We would be grateful if you could take time out of your busy schedule to discuss with us. We are very keen to engage in deeper discussions with the reviewers. **We further conducted experiments on AitW [1]**. We followed the environment settings of AppAgent and conducted comparative experiments on AitW. **As shown in Table below, Optimus-1 outperforms the PaLM 2, GPT-4V and AppAgent.** AitW is a popular, general benchmark that can demonstrate an agent's ability to operate apps in real-world scenarios. So the experimental results we provided below are sufficient to demonstrate Optimus-1's generalisation in real-world scenarios. We will include these results in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). Tab 1: Experiments on the subset of AitW. We report the partial match scores for AitW Standard split. | Method | Match Scores | | --- | --- | | PaLM 2 | 39.6 | | GPT-4V | 50.5 | | AppAgent | 52.4 | | Optimus-1 | **58.3** | We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. [1] Rawles et al. Android in the Wild: A Large-Scale Dataset for Android Device Control. 2023.
Response
Thank you for the response, and I appreciate the efforts for the new experiments. These have addressed my concerns and I'm raising my recommendation accordingly.
Response to Reviewer 397y
Thanks for your reply! We are pleased to have addressed your concerns. We will add these experiments in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).
Summary
Optimus-1 introduces an innovative Hybrid Multimodal Memory module that combines a Hierarchical Directed Knowledge Graph (HDKG) and an Abstracted Multimodal Experience Pool (AMEP) to address knowledge and experience management in long-horizon tasks.
Strengths
* Proposes a hybrid multimodal memory module, which includes HDKG and AMEP, offering an innovative approach to managing knowledge and experience in long-horizon tasks. * Experimental results demonstrate that Optimus-1 significantly outperforms all existing agents on long-horizon tasks and achieves near-human-level performance in many tasks. * The HDKG maps knowledge into a directed graph structure, enabling the agent to efficiently retrieve and utilize knowledge without needing to update parameters. * The AMEP summarizes not only successful cases but also failure cases, significantly enhancing the agent's learning effectiveness.
Weaknesses
* Improving long-horizon task performance through memory modules is common in LLMs; applying this directly to VLMs is not very novel. * Reaction time and decision efficiency might be issues; the current experimental results still show a large number of steps. * Although it performs well in the Minecraft environment, its performance in real-world applications, such as software manipulation and web navigation, has not been verified. * As new knowledge and tasks emerge, effectively updating and maintaining the knowledge (HDKG) remains a challenge.
Questions
See weaknesses
Rating
4
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes
Responses to Reviewer 3HCc
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows: > Q: AppAgent can also achieve the similar results with "Watching Demos". Would you like to explore the performance of Optimus-1 in AitW[1] and Osworld[2], it would be more challenging and persuasive. A: (1) **The experimental results in the Table below show that Optimus-1 outperforms AppAgent, even when compared to AppAgent with Watching Demos.** Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks. | Method | Success Rate | | --- | --- | | GPT-4 | 48.9% | | AppAgent-Auto. Exploration | 73.3% | | AppAgent-Watching Demos | 84.4% | | Optimus-1 | **86.7%** | (2) **We further conducted experiments on AitW [1]**. We followed the environment settings of AppAgent and conducted comparative experiments on AitW. **As shown in Table below, Optimus-1 outperforms the PaLM 2, GPT-4V and AppAgent.** It is important to note that our method is train-free, and due to time constraints, the experiments were conducted under insufficient “free exploration-teacher guidance” learning conditions. So it is unfair to compare it to baselines that fine-tuned on AitW dataset. We will include these results in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). Tab 2: Experiments on the subset of AitW. We report the partial match scores for AitW Standard split. | Method | Match Scores | | --- | --- | | PaLM 2 | 39.6 | | GPT-4V | 50.5 | | AppAgent | 52.4 | | Optimus-1 | **58.3** | (3) Since conducting experiments on OSWorld [2] requires a virtual machine and the environment configuration is quite complex, we are unable to provide experimental results for Optimus-1 on OSWorld in such short time. However, AitW is a popular, general benchmark that can demonstrate an agent's ability to operate apps in real-world scenarios. So the experimental results we provided above are sufficient to demonstrate Optimus-1's generalisation in real-world scenarios. We would like to provide experiments in more scenarios in future versions to demonstrate the generalisation of the proposed method. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. [1] Rawles et al. Android in the Wild: A Large-Scale Dataset for Android Device Control. 2023. [2] Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. 2024.
Response to Authors
Thanks for the experiments results. Would you like to share the implementation code and comprehensive trajectory records in an anonymous link, I will check with it and increase my score if the implementation is correct.
Responses to Reviewer 3HCc
Thank you very much for taking the time to discuss with us despite your busy schedule. We would like to share the code with you, but we have noticed that NeurIPS's rebuttal guideline clearly states, 'All the texts you post (rebuttal, discussion, and PDF) should not contain any links to external pages.' Therefore, we will discuss the feasibility of sharing the code link with the area chair, and we hope you can understand our concerns. Moreover, we would be grateful if you could focus more on our contribution of proposing the hybrid multimodal memory module and constructing the agent Optimus-1, which outperformed all powerful baselines in executing long-horizon tasks in Minecraft. This has been acknowledged by all reviewers (#3HCc, #397y, #mEzT, #AfEc, #wzp9). To address your concerns, we have done our best to verify the effectiveness of the proposed method on the AppAgent benchmark and supplemented experiments on AitW at your additional request. These experimental results demonstrate the generalization of Optimus-1 in general scenarios. We sincerely hope you can see the efforts and sincerity we put into addressing your concerns. We have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). Additionally, we are pleased to have addressed the concerns of other reviewers (#397y, #mEzT, #wzp9), and they have raised their scores. We would greatly appreciate it if you could take these aspects into consideration and give us a higher score. Thanks again for the time and effort you invested in the evaluation of our paper.
Responses to Reviewer 3HCc
We are still in contact with the area chair and have not yet received permission to share the code. We sincerely hope you can recognize the efforts and sincerity we have put into addressing your concerns. We would greatly appreciate it if you could take these aspects into consideration and give us a higher score. Thanks again for the time and effort you invested in the evaluation of our paper.
Thank you for taking the time and effort to address all questions. Things are clearer at this stage. There are a few follow-up questions. A1: > (1) We obtain teacher demonstrations from Minecraft Wiki. For each task group, we randomly select 5 tasks that are not included in the benchmark. We then create corresponding task plans based on content (synthesis relationship between objects) from the Minecraft Wiki a.) Could you please describe one simple example of how such a plan is constructed. Is an LLM used for this? Are these constructed by a human? A3: > (2) During the free exploration phase, Optimus-1 will randomly initialize the environment, materials, and tasks. a. )Does this happen after the initial expert demonstrations update the HDKG? b.) How does it happen without any initial demonstrations - what is the prompt, or few-shot example used? How is it constructed (human annotated)? --- Additional questions (AQ): a.) Do you mean by updating the HDKG, that a fact (i.e. triplet) is added to an actual KG? b.) Could you compare your method in more detail to works such as Voyager, specifically in terms of manual effort needed for the various phases and comparison of results with human effort vs. no human effort (especially on the harder long-range tasks).
Responses to Reviewer mEzT (1)
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows: > Q1: Example of how a plan is constructed during the teacher guidance phase. Is an LLM used for this? Are these constructed by a human? A1: **We obtain the plans required for the teacher guidance phase through an automated process**. For each task group, we randomly select 5 tasks that are not included in the benchmark. Taking the task “craft a wooden sword” as an example, We use a script to automatically obtain the crafting relationships for a wooden sword from the Minecraft Wiki: {1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword}, {1 log → 4 planks}, {2 planks → 4 sticks}, {4 planks → 1 crafting table}. These relationships can be represented as a directed acyclic graph. Then, by performing a topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2). Finally, we prompt GPT-4 to construct a plan in order from basic materials to advanced materials: {1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4. Craft a wooden stick 5. Craft a wooden sword}. The entire process is automated, simple, and easy to implement. We only need to construct 5 (tasks/group) x 7 (groups) = 35 plans in the teacher guidance phase, which takes approximately two hours. > Q2: Does the free exploration phase happen after the initial expert demonstrations update the HDKG? How does it happen without any initial demonstrations - what is the prompt, or few-shot example used? How is it constructed (human annotated)? A2: As described in **Section 2.3 in the manuscript**, we initialize the Hybrid Multimodal Memory as **empty and begin with free exploration to acquire basic knowledge**, such as crafting sticks and mining diamonds. We then proceed to **teacher guidance phase to learn advanced knowledge**, e.g., a diamond sword is obtained by a stick and two diamonds. The entire process does not require additional prompts, few-shot examples, or manual annotations. **In the free exploration phase**, we randomly initialize the environment, materials, and tasks. For the task “craft a wooden pickaxe”, we provide initial materials (three planks, two sticks), and then Optimus-1 (only the action controller activated) attempts to complete the task. If the environment feedback indicates the task is successful, the knowledge {3 planks, 2 sticks → wooden pickaxe} is added to the HDKG. Note that we randomly initialize materials and their quantities, which means that the task may not always succeed. As a result, each free exploration may not acquire the corresponding knowledge, but it can record the relevant experience (whether successful or fail). In the free exploration phase, **Optimus-1 learns simple atomic operations, such as crafting sticks in the Wooden Group and mining diamonds in the Diamond Group.** This phase is insufficient for Optimus-1 to learn advanced knowledge, such as crafting a diamond sword. **In the teacher guidance phase**, Optimus-1 executes each sub-goal sequentially according to the given plan. Once the task is completed, the materials and their corresponding relationships (e.g., {1 wooden stick, 2 diamonds} → {1 diamond sword}) are updated in the HDKG, and the multimodal experience of each sub-goal is stored in the AMEP. **Teacher guidance phase allows Optimus-1 to acquire advanced knowledge and learn multimodal experiences through complete long-horizon tasks.** > Q3: Do you mean by updating the HDKG, that a fact (i.e. triplet) is added to an actual KG? A3: **By updating the HDKG, triplets are added to an actual KG**. For example, the current KG contains (2 planks, 'craft', 4 stick), (wooden pickaxe, 'mine', stone). When new knowledge is acquired: (2 stick, 'craft', 1 stone pickaxe), (3 stone, 'craft', 1 stone pickaxe), (1 crafting table,'needed', stone pickaxe). These 3 triples added to current KG, then updated as: (1 planks, 'craft', 4 stick), (wooden pickaxe, 'mine', stone), (2 stick, 'craft', 1 stone pickaxe), (3 stone, 'craft', 1 stone pickaxe), (1 crafting table,'craft', stone pickaxe). **The rest of the responses are in the next comment**.
Responses to Reviewer mEzT (2)
This comment connects to the **Responses to Reviewer mEzT (1)** > Q4: Could you compare your method in more detail to works such as Voyager, specifically in terms of manual effort needed for the various phases and comparison of results with human effort vs. no human effort (especially on the harder long-range tasks). A4: (1) **Neither Voyager nor our Optimus-1 requires human annotation**. As stated in A1, we obtain the plans required for the teacher guidance phase through an automated process. This automated process is simple and easy to implement, and it takes approximately two hours. (2) As mentioned above, we do not require human effort to obtain the plans. So we cannot provide a comparison of results with human effort versus no human effort. (3) More detailed comparison between Optimus-1 and Voyager: Firstly, Voyager executes sub-goals in the Mineflayer environment by calling APIs (in the form of code), while Optimus-1 uses an action controller to generate low-level actions like a human, which is more challenging. Secondly, Voyager acquires knowledge through environment feedback during both learning and reasoning processes. For instance, when it fails to execute the task 'craft a diamond sword,' it acquires knowledge such as 'two more diamonds are needed' from the environment feedback. In contrast, Optimus-1 learns whether a task is successfully executed through environment feedback only during the learning phase and requires only a small number of plans to complete the learning process. Tanks for your discussions with us. If you have any further questions, please feel free to contact us.
Thank you very much clarifying your work. It sounds like excellent work indeed. Perhaps, some parts need further clarification and simplification. (Such as the initial plan acquisition step, at this stage it would be hard to replicate this step from your paper and discussion). Follow questions: 1. How could you illustrate the automatic plan acquisition step more clearly? What is required beforehand (eg a Wiki, in what format?) what are the exact steps (i.e. input and output of each component). 2. Does Optimus-1 exceed the capabilities of the teacher alone? How meaningful are these new capabilities?
Responses to Reviewer mEzT (3)
### Additional experimental results As you and some reviewers mentioned the feasibility of adapting Optimus-1 to other domains, we added additional experiments to demonstrate the generalisation of our method. We applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents. Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks. | Method | Success Rate | | --- | --- | | GPT-4 | 48.9% | | AppAgent | 73.3% | | Optimus-1 | **86.7%** | We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.
Thanks for the rebuttal
Thank the authors for the rebuttal. Thanks for pointing me to the comparisons in prior benchmarks. The rebuttal has clarified some of my concerns and I will adjust my score accordingly. However, there are still two important concerns remaining: ### The writing of the paper can be improved. As discussed above, the writing can be significantly improved. Currently the description of the technical section is very convoluted and can be better organized. Some of the terminologies are overly complicated, e.g. Hybrid Multimodal Memory, what's its difference with just Multimodal Memory? Abstracted Multimodal Experience Pool can be simplified to Multimodal Experience Pool. IMHO, such complication in terminology only makes it harder for readers to understand. Furthermore, it would be great to ensure the major claims in the paper are scientific as supported by the experiments. In particular, the experiments only show that it is important for the agent to know the rules of Minecraft (e.g. how are materials synthesized) and they are not enough to show GPT4V does not have structured knowledge. A significant effort is required for the work to be able to be published at top-tier ML conference. ### The application beyond minecraft remains unclear. In particular, a potential reason for HDKG to work well in minecraft is that there is a very clear game rule in minecraft (e.g. to make a stone axe we need stone and wooden sticks etc), and it is very unclear to me whether such designs can generalize to more realistic settings like real-world web navigation agents or robotics. I did read the response to reviewer 3HCc but I am not convinced that a better method for minecraft is interesting itself, unless it can be easily generalized to more realistic settings.
Responses to Reviewer wzp9 (1)
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows: > Q1: Currently the description of the technical section is very convoluted and can be better organized. A1: We appreciate your valuable and constructive feedback, which will be pivotal in enhancing the quality of our work. To present the methodology with clear logic, we first introduce the proposed hybrid multimodal memory module, followed by a detailed description of the Optimus-1 architecture. The hybrid multimodal memory is coupled with the operation mechanism of Optimus-1, while the current version lacks detailed descriptions of sub-goals, the reflection module, etc. We will add these descriptions in Section 2.1.1 to ensure logical coherence and avoid reader confusion. Additionally, in each subsection, we will organize the content in a top-down manner, following the sequence of motivation, high-level idea of the method, and implementation details. We will revise the manuscript based on your suggestion. We also note that the reviewers found the manuscript well-written and easy to follow (R#397y, R#AfEc), so these revisions will not affect the main contribution of our work. > Q2: Some of the terminologies can be simplified. (1) **These terminologies reflect the characteristics and innovations of proposed methods**. For example, the Hierarchical Directed Knowledge Graph links knowledge at different levels (wooden, stone, diamond) through **Directed** graphs, forming a **Hierarchical** knowledge graph, which differs from previous knowledge graphs. The Abstracted Multimodal Experience Pool **Abstracts** long-sequence multimodal historical information into multimodal experiences, whereas existing multimodal memory mechanisms do not summarize multimodal information. (2) **We can use abbreviations in the manuscript to describe these terminologies**. Although we introduced the Hierarchical Directed Knowledge Graph (HDKG) and the Abstracted Multimodal Experience Pool (AMEP) in Section 1 of the manuscript, we have frequently used the full names in subsequent sections out of concern that readers might forget or misunderstand the meanings of HDKG and AMEP. We will consider using abbreviations in the manuscript to describe these terminologies to facilitate easier reading for the readers. > Q3: Difference between Hybrid Multimodal Memory and just Multimodal Memory. A3: **Hybrid Multimodal Memory is different from existing multimodal memory**. As stated in the Introduction section of the manuscript, the hybrid multimodal memory module consists of structured knowledge (graphs) and multimodal experiences (text, image sequences). It stores multiple heterogeneous contents in a **mixed manner** and dynamically summarizes long-sequence multimodal information. In contrast, existing agents only store text and images as multimodal memory and do not summarize them. For example, Jarvis-1 [1] stores text and image sequences without summarizing multimodal information. Moreover, the comparison of memory mechanisms in existing Minecraft agents is shown in **Table 7 of the manuscript**. > Q4: Ensure the major claims in the paper are scientific as supported by the experiments. A4: Thank you for your suggestions. We will revise the claim in the Introduction section in the manuscript: “Existing Multimodal Large Language Models such as GPT-4V lack sufficient knowledge in Minecraft”, and results of **Table 2 in the manuscript** support this claim. **The rest of the responses are in the next comment**.
Response to authors
Thanks for authors' rebuttal. I would will change my score to 4. I look forward to seeing some experiments in software manipulation or web navigation based on Optimus-1 framework and also open to further modify the score.
Responses to Reviewer 3HCc
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows: Q1: I look forward to seeing some experiments in software manipulation or web navigation based on Optimus-1 framework and also open to further modify the score. A1: To address your concerns about the generalisation of our method in real-world scenarios, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents. Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks. | Method | Success Rate | | --- | --- | | GPT-4 | 48.9% | | AppAgent | 73.3% | | Optimus-1 | **86.7%** | We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. If you have any further questions, please feel free to contact us. [1] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.
Reply to Authors
Thanks for the additional experiments. Actually, AppAgent can also achieve the similar results with "Watching Demos". Would you like to explore the performance of Optimus-1 in AitW[1] and Osworld[2], it would be more challenging and persuasive. [1] Android in the Wild: A Large-Scale Dataset for Android Device Control [2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Thank the authors for the rebuttal
I have read the rebuttal and other reviews. Most of my concerns have been solved, so I will maintain my original score as 6: Weak Accept.
Decision
Accept (poster)