Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training

Summary

The authors propose an approach that takes advantage of human-object interaction videos to improve policies. Their proposal involves three steps: 1) learn a video tokenizer common to both human and robot videos. 2) Learn a discrete-diffusion model that performs video token denoising on both human and robot videos. 3) Learn a discrete-diffusion model that performs action denoising on only robot data. They show this technique results in better performance as compared to baselines.

Strengths

1. Proposed idea is very interesting. It brings together ideas from self-supervised pre-training and diffusion policies. 2. They achieve strong performance on a number of manipulation benchmarks outperforming many baselines. 3. Clear explanation of the methods.

Weaknesses

1. The paper claims that pre-training on a large dataset of human-object interaction videos without actions is useful for learning actions for robots. However from the experiments it is not clear how much the human videos are adding to the policy performance? There is a significant amount of data difference between the models trained with or without human videos. Yet the performance between the two models is close. How does the model without human data perform given some more robot trajectories? In Fig. 1, for comparison without human videos the drop in performance in marginal. In Fig. 5 seems there is a bigger drop in performance which might be compensated by having multiple camera views in the robot training data. 2. Ego4D videos are very different visually from the tasks and environments shown. It is not clear which tasks from Ego4D are helping in transferring good features for the robotics tasks. How much of the video tokenizer codebook is shared between the human and robot videos. Because the dataset statistics look so different I wonder if enough of the codebook is being shared by the two datasets? 3. Generating egocentric videos is challenging. If the model is good at generating egocentric videos then that in itself is a big contribution. What do the generated videos look like for the Ego4D dataset? Does performance on this task correlate with better downstream policy performance or is the fine-tuning step capable of recovering good policies on video prediction models that are not that good? 4. Another baseline to compare against would be a VideoMAE[1] kind of method to learn the video features on just robot dataset and then do diffusion on that. This will be a good way to make the case for a tokenizer based diffusion model to learn video features. [1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Zhan Tong, Yibing Song, Jue Wang, Limin Wang

Questions

See above weakness section.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes.

Summary

In this paper, the authors focus on the problem of policy learning via leveraging large video data without action labels. To this end, they employ a discrete diffusion framework which is first employed to predict future quantized video frames. Afterwards, the model is fine-tuned on robot data to learn the final policy. The authors evaluate their approach on the Meta-World and RLBench and demonstrate promising quantitative results over other approaches.

Strengths

1. The paper addresses a very timely problem: Most available video data has no action labels. A major question is how to properly leverage this data to learn general policies for control tasks. 2. For an approach which utilizes a pre-training and a fine-tuning stage with frozen networks, the approach isn't overly complex. It's simple relative to other approaches. It therefore should be easy for researchers to implement and reproduce results. 3. Quantitative results for both Meta-World and RLBench are strong over baselines. The results in are quite promising.

Weaknesses

1. Technical novelty is limited. The approach is similar to a lot of other approaches: quantize videos using VQVAE, predict future tokens using transformer, somehow account for missing action tokens. 2. The scope of domains for evaluation domains is somewhat limited. It would be interesting to see if the results hold on other robotic datasets.

Questions

1. While I believe the current results in the paper are convincing, it still may strengthen the paper to demonstrate results on even more datasets.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the authors have properly addressed limitations.

Summary

The authors propose utilizing large scale internet videos along with robot data to train a diffusion model which predicts video future frames conditioned on past video frames and language description of videos. This diffusion model operated on the vector quantized embeddings of the video frames. Subsequently, this diffusion model is adapted for action generation which is trained using a smaller amount of robot video data with action labels. The analysis is performed on the Metaworld suite and RLBench.

Strengths

1. The paper is well written, clear and easy to understand 2. The idea is well motivated. The question of utilizing internet scale knowledge for embodied intelligence is an open area of research. 3. The experiment results are statistically significant with multiple runs reported accompanied by error bars. 4. The method performs well in comparison to baselines.

Weaknesses

1. **Missing Baselines:** How does the method compare against other generalist imitation learning methods like Octo, RT-1 etc? It's unclear how such methods compare in the finetuning setting. 2. **Missing Ablations on amount of robot data used during pretraining:** While no action labels have been used during pretraining, the model still has access to robot trajectories. How does the method perform when no robot data is present during pretraining? 3. **Lack of robot experiments:** Does the model extend to any real robot setups?

Questions

1. What are the computational costs of training the backbone? 2. Are the results on RLBench and Metaworld finetuned for every task or are in multi-task setting, i.e., do you have one model that yields the aggregate result on Metaworld or do you have one model per task?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1. The method is currently limited to simulation and it remains to be seen if the results will transfer to real robots. 2. The method still requires 20 demonstrations per task during pretraining. It’s not clear if this is a kosher assumption. What if behavior cloning was used on these 20 demonstrations? Would this yield higher results? If so, its unclear why one would use VPDD?

Summary

The paper targets at transferring the knowledge from human videos to robotic manipulation policy. The paper proposes VPDD, a method that first pre-trains on video generation on both human and robot videos and then fine-tunes on robotic manipulation data with action labels. The intuition is, by learning to generate human and robot videos, the model obtains a "prior" (e.g., task planning, trajectory planning, etc.) of robotic manipulation which makes it easier to learn diverse manipulation tasks with even few-shot data. The authors compare VPDD to several robotic pre-training baselines and show non-trivial improvements over several benchmarks.

Strengths

Clear motivation that robot manipulation data is scarce and utilizing the large amount of human data will be helpful. The design of the model and the training pipeline is sensible and the results look promising.

Weaknesses

1. The pre-training stage is trained jointly on human and robot data. The amount of human data is pretty abundant here but how many robot videos are we using here? If the robot videos are too few, will it affect the resulting pre-trained model from this stage? If the pre-training stage still needs a lot of robot videos, then it still faces the problem of lacking enough robot data. On the other hand, if the pre-trained model can still generalize pretty well without robot data in the pre-trianing stage, i.e., pre-training on human data only and fine-tuning on a small amount of robot data can work pretty well, it would be very helpful. 2. Since the main message of the paper is training on human data can help learning robotic manipulation policies, it would be good to see how the performance changes when trained on different amounts of human data, for example, a scaling curve with x-axis being the number of human videos and y-axis being the performance. It would be interesting to see if the performance can keep going up by feeding more and more human data, or is there somewhere it saturates.

Questions

1. In Figure 4, VPDD w/o human videos is still better than other baselines. What's the reason here? Assuming VPDD-w/o-human has seen the same robot data as other baselines, what's the main factor that causes such a difference? 2. The authors tried VPDD on unseen scenes such as varying background and camera poses. What about unseen tasks? Assuming there are some tasks presented in human videos but not in robot videos, will the model learn to do these tasks, at least to some extent? 3. As a bonus, it would be good to evaluate some off-the-shelf pre-trained video generation models such as Stable Video Diffusion for future exploration, but I don't think it affects the main conclusions of this paper.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See above.

Thank you for the response

I would like to thank the authors for their thoughtful response. My concerns are all addressed and I increase my score to 7.

Respectful Reminder

Dear Reviewer, We hope this message finds you well. Since the discussion phase will end in approximately two days, we want to kindly follow up on the response we provided to your valuable comments and questions. Your feedback has been crucial in refining our work, and we greatly appreciate the time and effort you have invested in reviewing our paper. If you have any additional thoughts or questions about our responses, we would be more than happy to address them. Thank you once again for your insightful review! We look forward to your further feedback.

Response to Rebuttal

I would like to thank you for addressing my concerns. I stand by my rating. I believe this paper is an interesting contribution to the conference and would like to see it accepted.