OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents

Summary

The paper introduces a novel approach for behavior tokenization in agent domains, utilizing a unified token transformer. The key contributions include the development of a self-supervised behavior encoder that learns a vocabulary of actions. The generated discrete tokens for actions, augments vocabularies into MLMs for autoregressive modeling. The authors curate a large-scale Minecraft and multimodal QA dataset, including synthetic annotation generation, to train their model. The experiments demonstrate that the model surpasses baselines which go through a text bottleneck or directly map from pixels to low level actions.

Strengths

- The introduction of a quantized action codebook and the use of FSQ for behavior tokenization represent a novel approach to representing sub-goals, as opposed to text subgoals which may be limiting or require a predefined set. - The authors beat strong baselines DEPS and GROOT on long-horizon tasks in Minecraft, and show strong performance in open-ended instruction following and question answering. - The paper is generally clear and well-organized. - The proposed approach has potential applications beyond Minecraft, offering insights into behavior tokenization and hierarchical learning.

Weaknesses

- The impact of the new dataset compared to the proposed architecture is unclear. Further analysis is needed to isolate the effects of the dataset from the architectural innovations. - Given the focus throughout the paper on the architecture contribution, the paper lacks comprehensive ablations to validate the contributions of individual components, such as FSQ and curated dataset.

Questions

1. The paper proposes a new dataset and architectural methods. Can you clarify the individual contributions of each and their combined impact? 2. More comprehensive ablations to demonstrate the utility of the proposed architecture and the new dataset would strengthen the paper. Additionally, comparing against multimodal LLM training methods like QFormer or Perceiver to understand the advantages of the encoder approach and training stages. 2. The ablations in Table 6 need more detail. Does the language goal include memory and caption text? 3. How many episodes per task are used in Table 6? Why not ablate the full subtasks to provide a larger sample size? Additionally, why are ablations of the training phases and behavior tokenizer architecture not included? 4. Why are Jarvis-1 and Voyager not included in the comparisons in Table 2? 5. Can the codebook be used to derive interpretable skills?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors address some limitations, but additional suggestions for improvement include: - More comprehensive ablations to demonstrate the utility of the proposed architecture and the new dataset would strengthen the paper. Comparing against multimodal LLM training methods like QFormer or Perceiver could highlight the advantages of the proposed approach. - Ensuring consistent training data across baselines would provide a clearer comparison of model performance, or providing more experiments on the contributions of each component to back up paper claims.

Summary

This work presents OmniJARVIS, a instruction following agent for open-world Minecraft. The agent works by learning a behavior encoder that generates behavior tokens conditioned on textual, visual and action inputs via self-supervised learning at first stage, then a multimodal interaction sequence can be packed with the learned tokenizer and a policy decoder is trained with this tokenizer autoregressively with the objective of predicting action sequence directly. The proposed method is achieve good performance on atomic action tasks, significantly better performance on programmatic tasks comparing with baselines, and better performance on open-ended tasks where instruction is given creatively. Comprehensive ablation experiments are conducted on behavior tokenizers, input modalities, and vision tokenizer.

Strengths

1. The agent has a similar structure with GROOT, but replaced VAE-based approach with Finite Scalar Quantization (FSQ). 2. The agent demonstrated significant performance gain on proposed experiments over all baselines.

Weaknesses

1. Since this work get insights from GROOT, a slightly more comprehensive compare and contrast is preferred, especially, it appears the difference between these two works is not limited to how to learn trajectory representation, but also about input data format. 2. Writing could be improved: 1. The description for 2nd stage of training seems incomplete (line 175) 2. Typos in general 3. The description of how to handle longer trajectories (> 128) is not clear to me.

Questions

1. Is OmniJARVIS trained considering programatic task? If yes, in Table 2, do other baselines also trained with programatic task data? if not, If not, how do you handle this difference when creating those baselines? 2. Just to confirm, in Table 2, the baselines are retrain / finetuned with the data constructed in this work? 3. GROOT and OmniJARVIS have close evaluation result on atom tasks as in Table 1, but far apart results for programatic tasks, could you provide more discussion on the reason? Clarifying first two questions would be helpful for this question. 4. Figure 5. Left and right seems similar visually, more explanation and description on what those behaviors are would be helpful for understanding. I will consider raising score if above questions are addressed.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Authors provided a discussion of scalability of the agent: Scaling law is applicable for OmniJARVIS's instruction tuning process, the eval loss exhibit a log-linear decrease as data scale for tuning increases. Scaling up VLM improves performance but saturation is observed at 7B parameters.

Summary

This work introduces OmniJARVIS, which jointly reasons over visual observations, instructions, self-generated text, and actions. OmniJARVIS models actions via behavior tokens, which are discrete embeddings that are separately learned on a behavior dataset. A policy decoder converts these behavior tokens to a sequence of low-level actions. OmniJARVIS also has a pipeline for synthesizing instructions, memory to track what happened in a long trajectory, and chain-of-thought text from offline observation action data. OmniJARVIS outperforms baselines in Minecraft on atomic, programmatic, and open-ended tasks.

Strengths

1. To the best of my knowledge, the behavior tokenization is a novel way to connect VLA models to actions. This approach has the advantage that the large LLM model does not need to be run for every action generation. Instead, the lighter weight policy decoder can generate a sequence of actions. 1. The pipeline of using interleaved self-generated memory with a VLA is also novel. The paper shows the value of this additional data in Table 5 and presents a scalable way to generate it for Minecraft. 1. OmniJARVIS presents a way to scale end-to-end policies for complex long-horizon tasks like Minecraft. Prior approaches like Voyager, while able to operate on long-horizon tasks, assume primitives that can be called via language. Other methods, like STEVE-1, directly output keyboard actions but struggle with long-horizon tasks (as shown in Table 2). OmniJARVIS operates directly from pixel inputs and outputs keyboard and mouse actions yet can complete long-horizon tasks with a high success rate. 1. The paper shows extensive results in Minecraft with multiple tasks in 3 task setups of atomic, programmic, and open-ended tasks. In each setting, OmniJARVIS mostly outperforms the relevant baselines. 1. OmniJARVIS can scale to larger models and datasets, as demonstrated in Fig 4.

Weaknesses

1. The paper lacks many important details. Little detail is given about the encoder and policy decoder architectures (see (2) under the questions section). 1. Important behavior token ablation experiments are missing. The context length for the behavior tokens is never analyzed and only 128 is used throughout the paper. However, this could be an important setting for the behavior tokens. Additionally, the paper does not ablate the FSQ settings to determine the necessary codebook size for the behavior tokens. The behavior tokens are also only conditioned on observation sequences without actions (L92). However, this decision is never justified. 1. Experimental result details are unclear. On L215, the paper states it uses "30 programmatic tasks to evaluate the performance of different agents". But as far as I can tell, these 30 tasks are never described. 1. The value of including the question answering dataset for instruction following is not verified. It is possible including this additional training source is the primary cause of the OmniJARVIS outperforming baselines considering it constitutes a third of the examples. 1. OmniJARVIS performs worse than the GROOT baseline in collecting harder resources like wood and cobblestone (Table 1). However, I do not see this as a large weakness because OmniJARVIS is capable of programmatic tasks unlike GROOT. 1. It's unclear how the OmniJARVIS can be used beyond Minecraft. OmniJARVIS exploits that the OpenAI Minecraft data includes oracle meta information for synthesizing the instruction, memory, and thought. 1. The paper does not clearly discuss limitations. 1. Agent behavior and failure modes are not analyzed. See point (6) of my questions. Minor: 1. Table 2 should clarify what the numbers in parentheses next to the task type mean. I assume they are the number of programmatic tasks per category. 1. A checkmark representing the training setting is _removed_ is confusing. I suggest changing to an "x" mark instead.

Questions

1. In regards to weakness (6), how can the OmniJARVIS be applied to non-Minecraft domains? 1. What are the policy architectures for the encoder and policy decoder? How many parameters are there? How long are they trained for? 1. For the behavior tokens, why introduce new tokens into the LLM vocabulary as opposed to reusing infrequently used tokens in the vocabulary as in RT-2? 1. How does the paper arrive at 1T tokens on L187, given the previous token counts add up to 1B tokens? 1. In Table 5, how can OmniJARVIS work without the instruction? 1. What are the failure modes of OmniJARVIS? I am specifically interested in where OmniJARVIS fails in the programmatic tasks in Table 2. For the challenging task of Diamond, qualitatively, what behaviors does OmniJARVIS exhibit to succeed at this task?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

No, the paper does not clearly state its limitations.

Summary

This paper presents a Vision-Language-Action (VLA) model, OmniJARVIS, for open-world instruction-following agents in Minecraft. OmniJARVIS leverages unified tokenization of multimodal interaction data to enable strong reasoning and efficient decision-making capabilities. This work introduces a behavior tokenizer to encode behavior trajectories into compact representations that could be effectively modeled with other modality tokens via autoregressive transformers. The experiments are conducted in a variety of tasks in the Minecraft environment, along with some analysis of design choices and scaling properties.

Strengths

1. This paper is well-written and easy to follow. 2. Illustrations of the pipeline are informative and clear. The authors demonstrate their proficiency in pipeline illustrations in Fig.1 and Fig.3, which contribute to improving readability. 3. The proposed approach is scalable with respect to model sizes. As depicted in Fig. 4, a larger model results in lower evaluation loss, showcasing the scalability of this approach. 4. The authors greatly utilized established LLMs to construct a large-scale multi-modal interaction dataset. The proposed strategy of data collection could provide inspiration for future research in multi-modal embodied understanding.

Weaknesses

1. Incremental design with limited novelty. To my knowledge, the key idea of this work is tokenizing observations into compact behavior representations. However, the detail of the tokenizer works similarly to the previous work GROOT. The most notable difference between the two methods is that GROOT utilized continuous latent while this work uses quantized one (from lines 88-89). In my view, the idea of quantizing the action space seems incremental to the original GROOT and cannot adequately support the innovative quality of this paper. The authors are encouraged to add a separate section/subsection to detailedly discuss the difference between these two works. 2. Lack of training/inference efficiency analysis and comparison. The authors claimed the efficiency of this approach multiple times throughout the paper, such as in lines 102-104. However, there is no direct comparison of the actual training/inference speed/memory cost between different approaches. Some quantitative supporting evidence could be helpful.

Questions

Since the authors leverage GPT3.5 to augment previous datasets with more language labels, what is the price/cost for constructing such a large dataset with GPT APIs?

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

There’s no separate "Limitations" section in this paper. The authors are encouraged to point out that the proposed approach is only validated in the Minecraft environment, and the generalization and transferability to other scenarios are not explored in this paper.

Thank you to the authors for their comments

I thank the authors for addressing my clarifications and comments. My concerns have been mostly addressed with the additional experiments. Given the impact of the data used, I think the authors should be more transparent in the writing of the paper about the impact of the dataset versus proposed architecture, and how the training data differs from that used by baselines. I have raised my score to a 6.