iVideoGPT: Interactive VideoGPTs are Scalable World Models

Summary

This work studies the setting of planning and prediction from video world models. It proposes to use a GPT-style transformer world model that incorporates action and reward information in it's context and prediction pipeline. The model is further equipped with a novel tokenization technique based on VQGAN. Finally, the model is evaluated on three downstream applications, namely video prediction, planning and model-based RL. Ablations on the models and tokenizers scalability are conducted.

Strengths

Problem Statement * The problem of studying dyna-style video algorithms with foundational models is interesting and warrants investigation. Clarity * The paper is very well written and the language is clear * The paper follows a very concrete thread and is easy to follow * All plots and figures are well-formatted and easy to understand Related Work * The treatment of related work is done very nicely. The paper cites a total of 120 other works which is not often seen and thus formidable. * The work highlights older as well as recent work and positions itself well within the literature Experiments * The experiments contain a large number of baselines and comparisons that are useful to understand the capabilities of the model for downstream return maximization.

Weaknesses

Clarity * In some cases, the writing makes very strong statements that are very broad and hard to be supported by evidence. E.g. * L97 "This model can acquire broad world knowledge". This statement is not supported and it is unclear to me what "broad world knowledge" means. I believe concise language about the capabilities of our algorithms is important. Motivation * One issue with the paper is that some of the motivation is not quite clear to me. The paper argues that the provided world model has two benefits: It is interactive and scalable. * First, it is not clear to me what it means for this model to be interactive. If the condition for a model is to be action-conditioned, then the dreamer world-model should also be interactive. In fact, even much older world-models would be interactive [1, 2]. * Second, it is not clear to me that other models would not be scalable and the paper provides no evidence that they are not scalable. The text states in line 91 that dreamer lacks scability, however not citation or evidence for this is provided. For instance, dreamer uses state-space models (SSMs) and recent advances have built scalable SSMs [1]. Experiments and Claims * For the tokenizer, the text claims that it "decouples dynamics information from context conditions" L161. This seems like a strong statement that would warrant evidence to be supported. I believe qualitatively looking at a small number of image sequence is insufficient to support such a strong claim about composition. * Overall, the experimental results seem a little weak. * The model is outperformed on most tasks in the video-prediction setting. It is unclear why the one of the weakest baselines was used for the for action-conditioning comparison in section 4.1. * In the visual planning experiment, the performance exceeds baselines in only 2/7 tasks. * In the model-based RL experiments, the performance is mostly comparable to dreamer. There are cases where the paper argues that their suggested algorithm outperforms dreamer but given that the results are only reported over $5$ random trials, it seems that the results are within variance making this claim too vague. (It’s a little hard to tell because shaded regions are very transparent) * The benefits of scalability for the downstream control tasks have not been demonstrated. * The work argues that high perceptual metrics don't necessarily correlate with good control performance (App B.2) but the main benefit of scale seems to be improved performance on those metrics. * While the work shows in section 4.4 that scaling leads to lower validation loss, whether or not this is correlated with downstream performance is not demonstrated. I believe this point is important because we have to ask what is the marginal benefit of increasing computational complexity to train larger and larger models, if the application to downstream tasks does not benefit much from scale. Showing that larger networks and lower error lead to improved downstream performance is an experiment that might validate the need for scale. * Similarly, the need for specialized tokenization has not been argued for downstream applications. It seems a little detached from the goal. The main goal of the paper is to argue for scalability and that this tokenization technique is helpful. However, for the downstream applications, neither the planning nor RL sections demonstrate the importance of scaling or tokenizing. One experiment I can think of is to demonstrate that 4x4 tokenization is insufficient due to reconstruction quality this claim. The relationship between scaling and the novel tokenization is not analyzed either. Overall, I think the motivation of the work is not quite clear enough, the contributed algorithm provides minor improvements in the control settings and the claims for interactivity and scalability could be strengthened. * The text claims that the tokenization that is presented is more efficient but there is no experiment validating that this is true. A complex encoder structure is used and it is unclear whether this is actually more efficient. An experiment to validate this could measure total wallclock times on the same hardware. [1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao. arXiv preprint arXiv:2312.00752. [2] Action-Conditional Video Prediction using Deep Networks in Atari Games. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, Satinder Singh. Advances in Neural Information Processing Systems 28 (NIPS 2015).

Questions

Q1: What are the training times of your algorithm training and how do they compare to the runtimes of Dreamer? Q2: Can you elaborate on the term interactive and why other models are not interactive? Q3: What are the key differences between the proposed model and the MaskVIT model and to which of the difference do you attributed the improved action-conditioned prediction performance in section 4.1?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

4

Limitations

The paper contains an explicit section outlining limitations of the work.

Summary

This paper introduces a new architecture for an action-conditioned video (and reward) prediction model. First, a VQGAN converts context frames individually into tokens. Second, a conditional VQGAN converts future frames individually into tokens, conditioned on the context frames (using intermediate representations from the context encoder). The idea is that the future encoder can focus on encoding changes in the scene, since the remaining information was already extracted from the context, allowing for a lower number of tokens. Then, an autoregressive transformer is trained to predict the next (future) token and is optionally conditioned on actions and can optionally predict the rewards in a reinforcement learning setting. The model is pre-trained on action-free robotic manipulation videos and then fine-tuned and evaluated on video prediction, visual planning, and model-based reinforcement learning tasks.

Strengths

- S1: Encoding the dynamics information conditioned on context information is an intriguing idea for world models. - S2: The paper is clearly written and the visualizations are informative and appealing.

Weaknesses

- W1: The paper is missing quantitative comparisons of the computational efficiency. For instance, it would be interesting to see the differences in training/inference time when using the different tokenizers (Figure 8(c)). Furthermore, the authors state at the end of Section 2 that recurrent world models like Dreamer are not scalable, but I'm wondering how significant the difference is, considering the autoregressive next token prediction and use of multiple tokens per frame. - W2: The authors argue that iVideoGPT eliminates the need for latent imagination in model-based RL (Section 4.3 and Figure 5). However, there are existing world models (e.g. IRIS [1]) that learn behaviors using reconstructed frames. Moreover, Dreamer could also learn behaviors this way by using the reconstructions from the decoder (but I understand that this would be a different method). In short, I don't think that this is a feature that is novel or specific to iVideoGPT. - W3: The proposed model has a lot in common with IRIS [1], which learns a discrete autoencoder (VQVAE) and uses an autoregressive transformer for next token prediction. IRIS is briefly mentioned in the related work section, but I think the authors could emphasize the differences in more detail. Some typos: - 79: "aims" - 80: "maximize"? - 85: "history of $T_0$ video frames"? - 86: "needs to" - 107: $D_c(z_t)$ should be $D_c(z_t^{(1:N)})$? - Also in Eq. (1) it should be $D_p(z_t^{(1:n)}|o_{1:T_0})$? - 687: Table 2 -> Table 3 I am willing to increase my scores after the listed weaknesses have been addressed.

Questions

- Q1: Is the world model also conditioned on the actions from the context? In line 87 the condition only includes $a_{T_0:t}$, but in the sequence of tokens $x$, the context also includes slot tokens. This can also not be recognized in Figure 3(b).

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors addressed all limitations adequately.

Summary

The authors present a paper that attempts to utilize actionless and action conditioned trajectories to learn a large scale interactive world model. This model is subsequently adapted for robot manipulation tasks. It is evaluated on video prediction, visual planning, model based RL. The model training is tested primarily on Metaworld in the embodied setup. The experiments demonstrate the generalization capabilities of the model.

Strengths

1. The paper is well written, clear and easy to understand 2. The idea is well motivated. The question of utilizing internet scale knowledge for embodied intelligence is an open area of research. 3. The experiment results are statistically significant with multiple runs reported accompanied by error bars 4. The authors present a novel tokenize scheme which can benefit other video foundation models

Weaknesses

1. **Performance on Video prediction:** The performance for the visual planning is rather unsatisfactory. Do the authors have any intuition about why their method only outperforms the baselines in 2 of the setups? 2. **No analysis on amount of action data needed:** During the motivation of the method, the authors discuss coming up with methods that are able to learn from freely available videos. Robot data on the other hand is expensive. Thus, ablations on how much robot data is needed are necessary to understand the dependence of the model on robot specific data and whether or not the method is indeed benefitting from freely available human videos. 3. **Lack of robot experiments:** Does the model extend to any real robot setups? Currently, the only low level control robot experiments available are on 6 in which the model matches the performance in 3. 4. **No human user studies:** the paper only uses numerically metrics like Frechet Video Distance to judge the quality of generation. This metric does not always align with quality. The results of such a study would help bolster the quality of the work. 5. **Some missing Baselines**: FitVid does indeed have an action conditioned model and this model could be used for all the MBRL experiments. This currently missing.

Questions

1. Is the dreamer baseline in the paper trained on OpenX data also? If not then the comparison is not fair. 2. How do you ensure that the $z^(1:n)_t$ contain all the information about future frames and that there is no leakage between which tokens contain what information?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1. The evaluation suite is currently very small should be expanded 2. Its unclear how much the human videos from Something-something-v2 help the method. If they do not help the method significantly, then this diminishes the contributions of the paper significantly as this would make it a study on the application of transformer based world modeling on large robot datasets like OXE. Inspite of these limitations and weaknesses, I think the work makes interesting contributions and I would inclined to increase my score if the authors are able to address my questions.

Required Action: Please Respond to the Author Rebuttal

Dear Reviewer yh7R, As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed. The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns. Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process. Thank you for your time and dedication to NeurIPS 2024. Best regards, Area Chair, NeurIPS 2024