Summary
This paper presents a novel approach to leverage pretrained Vision-Language Models (VLMs) for training Reinforcement Learning (RL) agents by utilizing them to generate rewards. Specifically, the authors employ S3D, pretrained on HowTo100M, as the backbone and measure the dissimilarity between online agent experience and task demonstrations to serve as the reward signal for training task-specific RL policies. They demonstrate the effectiveness of their proposed method on the Metaworld Environment and the Franka Kitchen Environment.
Strengths
1. This paper addresses an important and widely recognized problem, which is the grounding of VLMs.
2. The idea behind the proposed method is straightforward and intuitive.
3. The author thoroughly examines the performance of the proposed approach from six different perspectives: alignment, text-to-reward mapping, video-to-reward mapping, generalization to out-of-domain videos, decompositional generalizability, and an ablation study.
Weaknesses
I have some major concerns about the experiment section of this paper. There are several results that are not convincing and can be quite confusing.
1. The domain alignment between the demonstration and text description, as mentioned by the author, is indeed unsatisfactory. For instance, the policy should learn to close the green drawer instead of opening it when the task descriptor is "open green drawer." Using this misleading reward can lead to misleading learning outcomes.
2. In the language-to-reward section (Section 4.2), the author only demonstrates the performance on three "easy" tasks, which is not sufficiently convincing. Additionally, it is unclear why the proposed method outperforms dense task rewards. The dense task reward can be considered an oracle, so it raises questions about whether the performance improvement is due to insufficient training epochs or simply because the rewards themselves are inadequate.
3. The qualitative results presented in Section 4.3 are challenging to comprehend. It is difficult to understand how the robot can open the cabinet without even touching it. While I can imagine what happened in between, the visualizations provided in the paper do not offer enough information to support a clear understanding of the process.
4. Regarding multimodal task specification, it is puzzling why RoboCLIP-video (with a different object) yields the best performance instead of RoboCLIP-Edit. One possible interpretation is that the robot is primarily learning push-like behaviors from a video demonstration. However, this raises concerns about the overall effectiveness and usefulness of the edit method in achieving the intended goals.
Questions
1. What is RoboCLIP-Video in Section 4.2?
2. In Section 4.6, how did you use 4 demos to generate one reward for training?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I appreciate that the author acknowledges the limitation of unstable fine-tuning in their work.