Learning Universal Policies via Text-Guided Video Generation

Summary

This paper introduces a novel approach to text-conditioned video generation task by treating it as a goal-conditioned RL, where the text is formulated as the goal and the multi-step image sequences in the video are the consecutive observations. The proposed method, termed Unified Predictive Decision Process (UPDP), aims to learn effective control strategies. Compared to traditional MDPs, the sequences in UPDP are determined by both the initial frame and the task description. This enables bypassing reward design and facilitates non-Markovian modeling. To model decision-making within UPDP, this paper introduces UniPi, which consists of two key components: i) a diffusion model planner and ii) a task-specific action generator. Empirical evaluations on various control tasks demonstrate the effectiveness of the proposed method.

Strengths

1. This paper is clear to follow. 2. The proposed method is effective and seems reasonable. 3. A large number of experiments showing performance gain. 4. Synthesized Frames are illustrative and interesting.

Weaknesses

1. While the high-level idea presented in this paper is intuitive and reasonable, the detailed descriptions of certain specific modules require further emphasis. For instance, the video-based planner $\rho(\cdot|x_0,c)$ is employed to generate image sequences (or videos). However, the role of the action in this context is not clearly defined. If the planner is unrelated to the action, it raises questions regarding how the action can influence the state. From the statement in lines 115-116 showing that "This design choice isolates planning decisions from action-specific mechanisms, allowing the planner to be environment and agent agnostic", it seems suggest that this module is indeed a forecaster rather than a planner. 2. Text-conditioned generation has gained recent popularity. Although the paper states that planning through video generation poses challenges due to the need for specific initial images to complete tasks, guided generation (conditioned on an initial image and a language description) is a common choice that has been extensively studied. Similarly, inverse dynamics is also a well-explored method that has been proven effective. Therefore, the combination of these techniques may not provide significant novelty from a model design perspective. 3. Some details in the paper may lead to misunderstandings. For example, in section 3.1, the author proposes tiling as a means to ensure trajectory consistency. However, two questions arise: i) Does tiling involve concatenating the generated frame with the initial image of the video or the ground-truth image at a specific timestep? ii) How exactly does tiling provide trajectory consistency? I recommend the authors extend this paragraph with more details. 4. Figure 2 appears somewhat irregular, as the widths of the noises applied to video diffusion are equal, whereas the noises applied to Temporal Super Resolution vary in width. Furthermore, the outputs of the video diffusion module also exhibit different widths. Ensuring consistency in the widths of these elements would enhance the clarity of the figure. 5. The disentanglement between the planner module and the policy network is presented as a strength of the model. However, it remains unclear how the policy network can impact the planner or interact with the environment in such cases. Section 3.2 attempts to explain this relationship, but the details provided are not easy to follow. It is hypothesized that the video planner solely provides synthesized frames for reference (denoted as $\bar{o}$). However, it is unclear whether, when the environment provides an observation $o$, the inputs for inverse dynamics should be $(o_t, \bar{o}_{t+1})$ or $(\bar{o}\_t, \bar{o}\_{t+1})$. Further clarification is needed in this regard.

Questions

1. The specific difference between the super-resolution and low-temporal-resolution architectures mentioned in lines 179-181 appears to be that the former employs the observation tiled as a condition, while the latter does not. It is unclear whether the super-resolution and low-temporal-resolution architectures refer to the hierarchical structure mentioned in lines 184-194. To enhance clarity, I suggest the authors restructure lines 174-183 accordingly. Alternatively, if these architectures are not the hierarchical structure, the authors should provide a clear explanation or visual representation, such as a model figure, to elucidate the architectures for improved readability. 2. In Appendix A.3, it is unclear whether "replicate future controls across different pixel locations" serves the purpose of concatenating state-action pairs. Further clarification is needed to understand the specific role and functionality of this mechanism. The other questions are above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As the authors showed, the video diffusion process is very slow, even with some fast sampling methods. It's not suitable for real-time robotic control for now, while might be an interesting domain to explore in future work.

Summary

This paper frames the sequential decision-making problem as a text-conditioned video generation problem. Given a text-encoded specification of a desired goal and the first frame with the initial configuration, a planner generates a set of future frames that depict planned actions. The generated video is then used to extract control actions. This framework facilitates leveraging pretrained language embeddings and large-scale internet videos to enable combinatorial generalization and knowledge transfer across diverse tasks.

Strengths

1) Casting sequential decision-making as an synthesis problem naturally allows us to leverage a growing wealth of existing research in large-scale image/video/language generative models that encapsulate valuable world models for robotics. 2) Strong experiments: both simulation and real-world tasks are included.

Weaknesses

1) Some overlap with prior work: the bulk of the framework seems to borrow heavily from Decision Diffuser[1]. Specifically, [1] also uses classifier-free guidance diffusion to generate a sequence of states, followed by some inverse dynamics modeling to identify the action to execute. The main innovations of the work are centered around substituting engineering components in prior works; e.g. replacing unconditional generation from something like Diffuser[2] with conditional generation [1] [Is Conditional Generative Modeling all you need for Decision Making?](https://arxiv.org/abs/2211.15657) [2] [Planning with Diffusion for Flexible Behavior Synthesis](https://arxiv.org/abs/2205.09991)

Questions

1) I wonder how much of the combinatorial generalization exhibited by the model is dependent on the quality of the text embedding. For example, as shown in [1], using something like CLIP language embeddings can make it difficult to capture more abstract notions like "left" or "right" relative spatial positioning, affecting combinatorial generalization in some of the tasks shown in Section 4.1. Could the authors include an sensitivity analysis on the T5 embeddings? 2) The authors claim this approach allows for more universality when it comes to environment diversity or reward specification. But it seems to me that the inverse dynamics model discussed in Section 3.2 would have to be retrained for different tasks anyways. How do the parameter/data scales for this (smaller) model compare to the diffusion component? 3) How does performance for combinatorial generalization in Section 4.1 scale compared to something like BC? i.e. is there is a ceiling to training inverse dynamics on generative data -- could the authors demonstrate how performance scales with number of examples compared to something like [1] or [2]? [1] [Programmatically Grounded, Compositionally Generalizable Robotic Manipulation](https://arxiv.org/abs/2304.13826) [2] [CLIPort: What and Where Pathways for Robotic Manipulation](https://arxiv.org/abs/2109.12098)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the authors have discussed some limitations in their concluding remarks.

Summary

The authors utilize the enhanced capabilities of text-guided image synthesis, a recent advancement in deep learning, to engineer general-purpose embodied agents capable of sequential decision-making. The proposed method involves using language instructions as inputs to a text-conditioned video generation model, specifically video diffusion. Actions are then inferred from the generated video using a learned inverse dynamics model. As agent learning from video play has been a well-established approach in reinforcement learning and imitation learning for complex tasks, translating language instructions into video for general-purpose agents can be viewed as a universally applicable strategy. This approach addresses the challenges of diversity and heterogeneity in agent environments, offering a broader, more versatile solution that can leverage also pretrained large-scale language and language-video models.

Strengths

This paper demonstrates how to harness the potential of text-guided image synthesis for agent learning, providing evaluations through multiple experimental scenarios. The paper is insightful, and it can offer several meaningful perspectives for readers interested in using LLMs and multi-modal pretrained models for agent learning.

Weaknesses

The proposed method in this paper can be seen novel, marking its originality, and showcasing insights on the utilization of multi-modal models. However, I also think that the method description in Section 3.1 falls short on illustrating clear technical contributions. It seems to only incorporate the existing diffusion model and the inverse dynamics model in the proposed framework. Despite the innovative nature of the broader concept, a more detailed explanation and comprehensive analysis regarding how to improve and generalize the proposed method can enhance the clarity and overall influence of the paper; e.g., comparing with language instruction-following agents with multi-modal capabilities, and comparing with skill-based hierarchical RL approach that generate latent skill sequences (that can be also translated into actions later).

Questions

Could the authors explain the benefits of the proposed method, specifically in comparison to other instruction-following agents that can use pretrained multi-modal models? If each video frame generated can be seen as sub-goal representations in the task, and each sub-goal is dealt by the inverse dynamics model, hierarchical planning in Section 3.1 seems to be critical for the effective handling of long-horizon tasks. Could the authors more describe the procedure of sampling videos? In line 215, the open-loop controller is chosen for computation efficiency. Could the author explain more? What if the stochastic environment has some randomness? Could the authors explain why Transformer-BC and TT achieve low performance in the experiments? In Figure 5, the adaptable planning scenario is not clear.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The conclusion has specified the limitations including computation load of video diffusion.

Summary

A general framework, the Unified Predictive Decision Process (UPDP), was proposed in this paper. It leverages images as a universal interface, texts as reward specifiers and an independent planning module for policy synthesis. Powerful diffusion model was adopted into the framework of UPDP to generate authentic future video frames and inverse dynamics was used to generate downstream policies.

Strengths

1. This paper is well written, clear and easy to understand. 2. A very intriguing introduction of UPDP framework. UPDP enables better utilization and knowledge transfer of large-scale generative pretraining which may demonstrate a broader impact in the future applications. 3. Extensive experiments demonstrate the effectiveness of the proposed method, including high-quality video generation, combinatorial generalization and multi-environment transfer.

Weaknesses

1: Diffusion model: The adapation of diffusion model to UPDP lacks novelty. Temporal superresolution and tiling the context frame are commonly used tricks in video diffusion model. Although this can be regarded as "simple yet effective", this paper can be further improved if authors can include more domain specific design into the instantiation of UPDP with diffusion models. 2. Metrics: For table 4, first there is a typo in line 302 saying "higher FID and FVD" while actually it's lower FID and FVD. Besides, FID and FVD are also not informative in this case because they only measure the distance between the distribution of generation and real data, which cannot tell if the model follow the text prompt correctly. Beyond training a classifier, a better metric could be measuring IoU of bounding boxes between groundtruth and generations, which are produced by a pretrained object detector, or human evaluation.

Questions

1. In Figure 7, are there images generated by the diffusion model or videos taken during the action excution? The reflection on the microoven in "Turn Faucet Left" row looks too consistent to be something generated by diffusion while this reflection is not shown in the input frame.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Authors talked about the potential improvement of generation speed for diffusion model part. In general, I think this is a solid and intriguing work.

Thanks for the rebuttal

I appreciate the author's rebuttal, which addresses some of my concerns/confusion. Yet, I still think the novelty is not strong enough to have an "accept". In sum, I will raise the score to "6".