Grounding Multimodal Large Language Models in Actions

Summary

This paper studies how to "ground" multimodal LLMs (MLLMs) to the action spaces of agents with various embodiments. The authors examine "Action Space Adapters" (parameterization strategies; ASAs) for various embodiments and MLLMs and identify principles for constructing ASAs based on the target action space. The authors consider 5 embodied AI environments (3 continuous and 2 discrete) and over 100 tasks.

Strengths

The paper addresses an important issue for the future generalization of MLLMs to embodied environments and different action spaces. The authors conduct a variety of experiments in different environments and action spaces. The motivation for the method is given a theoretical grounding, and the experimental results provide concrete recommendations for adapting MLLMs to novel action spaces and environments. The combination of theoretical motivation and actionable takeaways is a strong contribution.

Weaknesses

Overall the paper is very dense. This is a complex topic so understandably there is a log packed into 9 pages and as a result there are many parts of the paper that are hard to follow. There are a fair number of parts of the paper that are not clearly written, which impacts my scores. If the authors' rebuttal clarifies these points satisfactorily, I would be willing to reconsider my evaluation. The major points are given in Questions. Some other issues are: Some terms are not clearly defined for the reader. For example, "codes" as used in Figure 3. (Again, if I simply missed where this was provided, please point it out). Similarly for "codebook". No specific example outputs are provided. Some examples of environments are provided in the appendix but there's no clear discussion of, for example, adaptations that the best performing methods get right or wrong. Providing these would help clarify the definitions issue and make the problem much more grounded (ironically) than the plot and charts in the paper currently allow.

Questions

1) Can you be clear about the difference between SemLang and Lang? The way it is written right now it reads as if the only difference between the two is that there are different numbers in the sequence being predicted. But since SemLang is predicting tokens that correspond to words, I assume that there is some underlying embedding that when softmaxed produced the token index. In that case, where to the numbers in the vocabulary of Lang come from and how are they being predicted if it's not using an underlying semantic (embedding) representation like SemLang? 2) Section 3.3: "In the complex environments we consider, the zero-shot performance of MLLM is poor across all ASAs, even with detailed prompts." Is this demonstrated anywhere, either in this paper (in the appendix perhaps?) or elsewhere (if elsewhere, please provide the citation)? 3) "We train with supervised finetuning for CALVIN, Meta-World, HabPick, and BabyAI. We train with reinforcement learning on Language Rearrangement." Can you please explain why this does not result in an invalid comparison between the environments, if they are trained for each environment using a) different methods and b) to the specifications of the environment themselves? Does this not risk overfitting the ASA to the environment and the training method? Or am I misunderstanding the goal here? 4) How precisely do the differences in embodiments affect the results? Are the embodiments those provided out of the box in each environment or do you do any adjustment or controlling for the effects of the differing embodiments?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The only limitations the authors mention is that they use only a single MLLM and the data collection requirements on RVQ and SemLang. These are good to mention but does not constitute a serious discussion of limitations IMO. There are differences in the way the models are trained/fine-tuned (see Q3 above) that complicate the axes of comparison, as do the wide variety of experiments conducted. Some discussion on the limitations of the analysis would also be welcome.

Summary

In this paper the authors present a way to adapt a Vision and Language model to perform action execution tasks in embodied environments. Specifically, systematically evaluate different ways of predicting actions both on task having both discrete and continuous action spaces. Thanks to this evaluation, it is possible to assess which are the most performant action prediction losses to use for different use cases. According to the authors' results, approaches based on VQ-VAEs are able to obtain the best performance in many tasks.

Strengths

1. One of the first papers that finally sheds light on the different approaches to performing action prediction in embodied environments. This is the most important contribution of this paper and I believe it will be really useful to refer to this set of experiments for future research. 2. They propose a VQ-VAE variant to generate latent codes for encoding actions. These codes can be intended as a way to learn "latent bins" to cluster the action space. Additionally, they propose a variant of this model based on the RVQ-VAE architecture to model a set of codebooks that are used to generate more precise actions.

Weaknesses

1. The VQ-VAE variants are indeed really interesting and novel. However, I find the description of this method a bit unsatisfactory because it omits some details regarding "how" you train these models. See my question below for details. 2. The authors chose a good set of tasks for their evaluation however I believe that a benchmark that would have been perfect for this work is VIMA-Bench because of its focus on systematic generalisation. CALVIN somehow offers this but I don't think it's as systematic as VIMA-Bench. 3. For discrete action spaces they use BabyAI however they only test with a very limited size of the grids and with few tasks. Please see my questions related to this point as well. 4. Some related work missing: - Team, Octo Model, et al. "Octo: An open-source generalist robot policy." arXiv preprint arXiv:2405.12213 (2024). - Pantazopoulos, G., Nikandrou, M., Parekh, A., Hemanthage, B., Eshghi, A., Konstas, I., ... & Suglia, A. (2023, December). Multitask Multimodal Prompted Training for Interactive Embodied Task Completion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 768-789).

Questions

1. I believe that a reader would appreciate the details of how you have arranged the dataset to train the VQ-VAE variants. The appendix contains some information but it doesn't fully address the nature of the data (e.g., "What is each example?"). 2. Why did you decide to follow the work from Carta et al. considering that it uses only a subset of the trajectory instead of the BabyAI benchmark which contains a range of different tasks of different complexity? Additionally, when you evaluate, are you sure that 100 episodes are enough to experience different configurations from the ones you used at training time? 3. Could you please clarify what is the language instructions for environments that do not have one such as Meta-World?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledge the limitations of their experimental setup rather than the limitations of their work more broadly. Specifically, I would recommend the authors consider the societal impact of developing embodied AI models. For instance, it would be important to acknowledge that these models are still far from being deployed in real-world scenarios considering that their performance, even on simple grid-world-like environments, is barely over the 30% success rate bar.

Summary

The paper empirically studies how to properly ground MLLMs into embodiments, with a particular focus on the action representations, including the continuous and discrete solutions. The authors conduct a thorough study on 7 methods, encompassing over 114 tasks. The research indicates that for continuous actions, optimal results are achieved by learning action tokens that precisely represent the action distribution (RVQ). For discrete actions, superior outcomes are obtained by aligning actions with the original semantic token space.

Strengths

1. How to properly ground MLLMs in embodied tasks is important and under-explored. 2. The paper is the first to systematically and comprehensively study the optimal recipe for action tokens. 3. The conclusions drawn from the empirical studies could provide guidance for subsequent research.

Weaknesses

1. All experiments are performed on LLaVA and LoRA. Conclusions may not be applicable to other MLLM architectures or scales. 2. Based on existing conclusions, the quality of the paper could be further enhanced if the authors could suggest whether to use continuous or discrete methods, how to break through the accuracy limit of current methods, or where future work should focus on improvements.

Questions

Please see the Weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed limitations in Sec. 5.