VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions

Summary

This work introduces VLMimic, a framework to acquire robotic skills by imitating from videos. The system uses VL foundation models to ground human-object interaction videos. Hierarchical representations are used to learn robotic skills and recorded in knowledge bank. In unseen environments, the skill adapter iteratively compares to transfer the skill. In extensive experiments, VLMimic outperforms baselines on the RLBench, as well as comprehensive results in real-world manipulation tasks.

Strengths

It's a strong work. 1. VLMimic not only learns high-level action planning but also learns fine-grained low-level actions, a feature not possessed by past works. 2. The proposed approach looks novel and intuitive. 3. The experiments are impressive. VLMimic outperforms on RLBench simulation. Real-world experiments are conducted to solve long-horizon tasks. Qualitative videos are provided.

Weaknesses

major: 1. Can you predict the system scalability? What's the memory cost of the knowledge bank and the speed cost of iterative comparisons when scaling up? Is it possible to process scaled datasets like Ego-4D? minor: 1. Some definitions in Line 151 are vague to me. What's the keypoint value? What's the visualized interaction? How do you do the rendering? I suggest some revisions in this part. 2. Inconsistent descriptions: 'high level' and 'high-level'

Questions

I am interested in real-world failure cases of your system, in the cases that can't be corrected by failure reasoning. Which is the main challenge, high-level or low-level?

Rating

7

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

Discussed in Appendix B.

Summary

The paper proposes a system that can perform imitation learning based on human demonstration videos. The authors design a system using many pre-trained components like VLMs and hand and object tracker and 3D reconstruction and pose estimation tools. They show that by combining these existing tools they can achieve compelling success rates on multiple robots.

Strengths

1. Impressive performance across real robots and multiple tasks. 2. Ablation studies showing importance of different components. 3. Detailed analysis of runtime of each component.

Weaknesses

1. The paper and the abstract give a lot of credit to "VLMs". However there are a lot of other learnt modules like SAM-Track, BundleSDF and FoundationPose in the system. A lot of hand-designed priors like 3D projection, grasp prediction etc are also baked into the system. These should also be given due credit along with VLMs. 2. The paper seems to be more of a robotics systems paper where many modules are connected to perform a task than a machine learning paper.

Questions

1. Which models are actually used for the system? The authors mention multiple models many times in the paper: VFM [34; 35; 36] , SAM-Track [37; 38; 39; 40; 41]. Which of these are actually used? 2. What is the knowledge bank used in the paper? Examples of how it is constructed and accessed? 3. How many times is each method run during evaluation? How much scene randomization is done between these episodes and how different are they from the training demonstrations?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes

Summary

This paper introduces a novel paradigm named VLMimic, which employs Vision-Language Models (VLMs) and multiple vision tools to ground object-centric information from demonstrations into fine-grained actionable skill sets. VLMimic exhibits a remarkable success rate in manipulation tasks and demonstrates significant generalization capabilities, learning effectively from a minimal number of human and robot videos.

Strengths

1. The proposed method performs well even with very few demonstrations (less than 5 videos). 2. The evaluations are substantial, encompassing both simulation and real-world experiments, with diverse tasks conducted and generalization of multiple aspects (position, intervention, cross-embodiment and cross-task) verified. 3. The writing in general is well-organized and easy to follow.

Weaknesses

The proposed object-centric manipulation pipeline encompasses multiple foundation models / vision tools to generate various intermediate representations (object detection results, object 6D poses, hand mask and trajectories, and task description from VLMs). This integration relies heavily on human prior knowledge to string all these models, rendering the process lengthy and complex. Moreover, this approach may only be adaptable to a limited set of interaction objects (for example, whose 'affordances', or 'object properties' as proposed in the paper, can be easily defined and recognized). *[Minor]* According to line167-168, the semantic constraints are identified based on both visual interaction $I_{v}$ and sub-task description $T_{\tau}$. But only $I_{v}$ is incorporated in Equation 3.

Questions

1. In the iterative comparison process, how does VLMimic determine which object, from the extensive knowledge bank of diverse objects, is utilized as the reference for guiding the adaptation of grasping and manipulation constraints? 2. Has the potential for cumulative error in the proposed modular design been taken into account? Specifically, is there a risk that an error at any stage of the process could ultimately lead to the failure of the manipulation task?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Possible limitations are discussed in the paper.

Summary

This paper proposes a method to learn a policy from human videos utilizing advances in vision-language models. The authors first parse human videos into several keyframes, then detect and track hand-object interactions. From the segmented interactions, they learn low-level actions by following the 3D hand-object trajectories in each segment. Additionally, the authors propose an iterative comparison stage to adapt to new scenes or objects. The method achieves better performance compared to other visual imitation learning algorithms in both simulation and real-world evaluations.

Strengths

This paper proposes a novel usage of VLM for imitation learning. The perception module is well-designed, and the ability to learn skills from perceived trajectories is particularly interesting. The paper presents comprehensive evaluations in both simulation and the real world. It also includes detailed ablation experiments on different design choices of the proposed system. It demonstrates superior performance over previous methods, especially on unseen objects and tasks.

Weaknesses

While this paper is well-written and achieves good performance, I have a few concerns regarding some claims and evaluations of the proposed method. First, the paper focuses on fine-grained action imitations. However, I’m not convinced that the skills demonstrated in the paper are fine-grained or distinct from previous work that uses VLM for planning. Specifically, the skills in this paper primarily involve grasping and moving along certain trajectories, without complex interactions (actions requiring precise manipulation such as insertion or grasping that utilizes the environment). On the other hand, similar skills can also be learned using the 'VLM as Planner' framework. For example, the 'Code as Policy' approach can perform tasks such as writing with pens. This skill is quite similar to the 'Spread sauce' motion described. Therefore, the claim that 'Fine-grained motions' cannot be learned in the 'VLM as Planner' framework needs further explanation. The proposed method seems to categorize motion skills as 'grasping' and 'manipulation'. If the point is that 'VLM as Planner' requires each skill to be defined manually while the proposed method can automate this process, it would be beneficial to clarify that the proposed method still defines motion skills, but in a broader or more generalizable manner. I think the performance of the baseline imitation learning algorithms is surprisingly low. Given 100 demonstrations, R3M-DP and DP should be quite successful, at least in simpler tasks such as Reach Target. This assessment is based on the original paper where they achieve a high success rate in more difficult tasks, as well as my empirical experience. Could you clarify whether the evaluation protocol was changed to make the task much harder?

Questions

The proposed system is rather complicated. While I appreciate the authors' efforts in assembling such a complex system, it is honestly a bit difficult to understand. I have detailed a few technical questions below. In the grounding module: * How are the keyframes obtained? * How does the language model identify the important object to detect (e.g., pie, plate, brush, and bowl)? * How are candidate action primitives (such as grasping and manipulation) determined? For skill learning: * Can you provide some examples of the Multiple-choice QA? * How are the keypoints obtained? I appreciate the evaluation on unseen scenes and objects. However, how similar do the camera viewing angles need to be? I assume there must be some degree of robustness because the perception operates in 3D, but is there any evaluation of this aspect? As mentioned in the previous section, the results for R3M-DP and DP are surprisingly low. Have you tested the implementation on other tasks (such as those reported in the original paper) to verify the correctness?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors report limitations in the supplementary material. The limitations section is comprehensive, though I would suggest moving it to the main text.