VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

Summary

This work proposed ICAL, aims to improve decision-making in large language and vision-language models by generating optimized trajectories and language annotations from noisy demonstrations and human feedback. ICAL abstracts noisy trajectories into optimized sequences with language comments, refined through human feedback during execution. The experiment shows ICAL significantly improves performance in benchmarks like TEACh, VisualWebArena, and Ego4D, surpassing state-of-the-art methods.

Strengths

1、Improved Performance: ICAL significantly enhances decision-making and task success rates across various benchmarks, such as TEACh, VisualWebArena, and Ego4D. 2、Reduced Reliance on Expert Examples: The method minimizes the need for expert-crafted examples by generating useful abstractions from sub-optimal demonstrations and human feedback. 3、Versatility: ICAL is effective in multiple domains, including dialogue-based instruction following, multimodal web tasks, and video action anticipation. 4、Human Feedback Integration: The approach incorporates human feedback to refine and adapt abstractions, improving the agent’s performance over time.

Weaknesses

1、The method still relies on human feedback for refining abstractions, which may not always be feasible or scalable. 2、The process of generating and refining abstractions heavily rely on GPT4-V 3、The effectiveness of ICAL is constrained by the capabilities of the underlying Vision-Language Models (VLMs), such as GPT-4V. 4、The method’s performance can be affected by the quality of the initial noisy demonstrations and the accuracy of human feedback.

Questions

The VLM-driven Abstraction Generation component appears to heavily rely on the performance of GPT-4V. There is no ablation study for replacing GPT-4V. If such results exist and I missed them, please indicate where they can be found.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper proposes a pipeline for Large Language and Vision-language models (LLMs and VLMs) to digest and learn from sub-optimal demonstrations and human feedback. The LLM/VLM, given sub-optimal task demonstrations, is prompted to produce abstractions of the trajectory (including task and causal abstractions, state changes, task decomposition and subgoals, and state abstractions), and possibly refine the given trajectory with insight from these abstractions. Optimally, these produced abstractions can be further refined by interactions with humans. Experiments show that the proposed framework greatly surpasses zero-shot CoT baselines and is competitive with/slightly better than having expert demonstations.

Strengths

- The authors provide sufficient details with their experiment environments and experiment setup, as well as the code, for readers to better understand and reproduce the results. - The paper proposes a novel method for LLMs to autonomously refine sub-optimal trajectories and build a high-level abstract understanding of the task based on sub-optimal trajectories. For very complex tasks, these high-level abstract descriptions have the potential to help humans better understand the task and the general strategies to accomplish the task. - The experiments span three different types of environments, demonstrating the generality of the proposed method. The proposed method also achieve good experimental results over the baseline methods.

Weaknesses

- The experiment result focuses on the accuracy of the method; not enough comparison is provided in terms of the efficiency. For example, whether is might be more efficient for the human feedback provider to directly edit and improve the sub-optimal trajectories and provide the human-refined ones to the LLM/VLM? - The scaling capability of the proposed method is unclear. In the proposed method, each trajectory needs to go through the human-in-the-loop fine-tuning process which seems quite inefficient. For simple tasks in the TEACh benchmark the method needs ~100 trajectories to perform well according to Figure 5. For more complex tasks it might be less sample efficient and difficult to scale. - It also requires the environment to automatically reset itself, which is another limitation. - Some text in figures 1 and 2 are too small and difficult to read and understand. A more illustrative example of what the task is, what are the inputs/outputs, and what kind of information is included in the abstracted state is desired.

Questions

- Can we easily design a reward function that specifies how sub-optimal a trajectory is? If we provide these sub-optimal trajectories and their corresponding rewards to the LLM/VLM, can it use this reward information to further refine the trajectories? The "in-context RL" capability of LLMs are previous studied in works like [1]. - What is the difference between expert and non-expert feedback (mentioned in lines 281-282)? Examples with qualitative differences can be very informative. [1] Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. https://arxiv.org/abs/2309.04658

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors fail to include sufficient details on the human-in-the-loop fine-tuning phase. The evaluation might be biased if the human feedback provider is very familiar with the task and the LLM; they might be able to provide much more effective feedback than non-experts. If this is the case, then the amount of training required for the human to provide more effective feedback is worth investigating.

Summary

The paper proposes ICAL, In-Context Abstraction Learning, which builds a memory of suboptimal experiences that are abstracted into states and plans, as well as correction and reflection from human feedback. The approach is based on extensive prompting to elicit structured representations from past experiences, and apply RAG at test time. ICAL exhibits improvement in success rates across virtual domains compared to raw VLMs and CoT methods.

Strengths

The idea of using extensive prompting to generate structured representation for RAG is clean and intuitive, and the authors clearly detail the steps involved. Using RAG enables the overall pipeline to improve continually.

Weaknesses

My concern with the paper is the limited contribution in terms of the approach. It is a relatively simple adaptation of previous methods such as CoT, ReACT, Socratic Model [1] as well as RAG. [1] Zeng, Andy, et al. "Socratic models: Composing zero-shot multimodal reasoning with language." arXiv preprint arXiv:2204.00598 (2022).

Questions

Could you further discuss the limitations of the work? I think currently the paper lacks thorough discussions on the limitations. For example, what are the common failure modes? And is there a viable and scalable solution to it potentially?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The overall prompting strategy is quite sensitive to the model performance. The fine-tuning improvement is quite limited as discussed in 4.6.

Summary

The goal of this paper is to teach VLMs novel tasks by prompting VLMs to create multimodal abstractions for unfamiliar domains. Given instructions paired with noisy demonstration trajectories, this paper proposes a method to encapsulate the information from these into examples consisting of optimized trajectories paired with generalizable language abstractions. A VLM is specifically prompted to produce abstractions such as essential task steps, cause and effect generalizations, expected state changes, a step by step plan, and relevant parts of state information. The abstracted example includes an executable trajectory which is executed and potentially refined with a human in the loop via natural language feedback. These examples are then used for in context of a VLM to improve task performance. The paper includes experiments on 3 datasets - TEACh, VisualWebArena and Ego4D, comparing respectively to the HELPER model - a prior SoTA model for TEACh, and GPT4V for VisualWebArena and Ego4D. The most improvement in seen in VisualWebArena, followed by partial success on TEACh. On Ego4D, performance is on par with supervised learning methods that use more data but Zero shot CoT with GPT4V performs better.

Strengths

The overall goal in this paper of using VLMs to distill demonstrations into canonical examples is an interesting combination of currently popular foundation models with more classical ideas such as case based learning. The paper includes experiments on multiple benchmarks with sufficiently different tasks to suggest generalizability of the proposed method. It is also refreshing to see that the authors do not hesitate to include the unusually good performance of Zero shot GPT4V on Ego4D which might be seen as a negative result. Given the topic of the paper, a number of engineering details and information about prior work is needed to fully understand the paper and increase the likelihood of reproducibility of the work. The authors have made a very strong attempt at this with a detailed appendix.

Weaknesses

While the appendix goes a long way towards reducing this, it is likely that the paper might still be difficult to follow for readers unfamiliar with the datasets and prior methods referenced in this paper.

Questions

It is possible I missed this in the paper, but I how was the human in the loop feedback obtained for the TEACh experiment? Given that the checklist says no human subjects were recruited, I assume this feedback was provided by authors. Is this correct?

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors include a discussion of limitations and potential negative social impact.

increase rate to 5

Thanks to the authors for the detailed response. My concern about the dependency on human feedback has been addressed. So I increase my rate from 4 to 5. I do not give a higher rate at the current stage. That is because I am still concerned about the generalization and reproducibility of the overall flow, as it heavily relies on the close-sourced GPT-4. If the proposed method can not help any open-source models, it is hard to justify that the contribution can be generalized to other VLMs.