H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation

Summary

The paper works on the problem of visual RL dexterous manipulation with pre-trained representations. Specifically, the authors propose to fix the pre-trained Convolution layers and adapt only the BatchNorm layers on a small amount of in-domain data, and then learn the policy based on the pre-trained feature with EMA updates. Experiments are performed on 12 simulation tasks.

Strengths

The idea of leveraging large-scale pretraining to boost the performance of visual RL is interesting and well-motivated; The proposed method for adapting pre-trained feature for dexterous manipulation is simple and straightforward; The paper is well-written and easily read.

Weaknesses

For some tasks, the proposed method seems on par or slightly worse than baselines (e.g., Door, Relocate Mug, Relocate Mustard Bottle); The necessity of stage 3 in the pipeline can be called into question since the performance of the pipeline without stage 3 appears to be comparable to the performance of the full pipeline in the Hammer task. No limitations or failure modes are presented in the paper.

Questions

In Fig. 6 (b), why would larger or smaller momentums lead to better performance than an intermediate one? Could authors give more theoretical analysis on this? It seems like the standard BatchNorm without EMA already achieves comparable performance. It would be great if more theoretical guidance on choosing the momentum value is provided instead of using empirical study; Further clarification or analysis from the authors regarding the significance and contribution of stage 3 would be valuable to better understand its role in achieving improved results; In Fig. 8, it would be better to also present the scales for X-axis and Y-axis.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

No limitations or failure modes are presented in the paper.

Summary

The paper proposes H-InDex for dexterous manipulation tasks with a 30 DoF robotic hand in simulation. The method consists of three stages: the first stage adopts a pre-trained network that is trained with a 3D human hand pose estimation dataset; the second stage adapts the representation with self-supervised keypoint detection, which updates only 0.18\% parameters of the entire model; and the final stage learns an RL policy on top of the adapted module while also adapting representation by using the exponential moving average for changing distribution based on training state. The method is extensively evaluated on 12 different visual dexterous manipulation tasks and is demonstrated it outperforms the current SoTA baselines by a 16.8\% absolute improvement and also achieves superior sample efficiency in 10 out of the 12 tasks. The authors also tested the effectiveness of the proposed human hand prior by comparing the method with several different priors and showed the hand prior from human movies does improve the performance. Finally, ablations are conducted to show the effects of each state, how the adaptation works, qualitative results on keypoint detection, and visualization of the adaptation in BatchNorm layers. Overall, the paper is very well written and organized, and the reviewer enjoyed reading this work. However, there are some unclear descriptions and conclusions that cannot be drawn from the experiments. The reviewer would be happy to raise the score if the authors could answer the questions and/or clarify the reviewer's misunderstanding.

Strengths

- The idea of using a prior from human hand pose estimation is interesting, and the experimental results support the soundness of the method. - Only adapting the limited parameters by changing the statistics of the BatchNorm layers is also interesting and could be useful for other tasks that involve domain transfer, such as sim2real. - The experiments are extensively conducted and they support the idea of the proposed method. However, there are some unclear descriptions and/or conclusions that are summarized in the Weaknesses section.

Weaknesses

- The authors draw some conclusions that are not empirically supported or require more experiments. The reviewer thinks the authors need to tone down their claims and/or cite adequate prior works to justify the claims. For example, - L.145-146: 'While finetuning only a small portion of parameters, it empirically outperforms both a frozen model and a fully finetuned model.' Please show the numbers probably in the appendix. - L.209,212: 'with appropriate domain knowledge, ConvNets can still outperform ViT', 'these observations highlight the task-dependent nature of the strengths exhibited by ConvNets and ViTs.' The reviewer thinks these are rather the nature of algorithms, not feature extractors (which is also just a hypothesis, without any evidence). To draw this conclusion, the authors need to do experiments with the same backbone networks, the same algorithms, and large enough amounts of random seeds. Without those, the reviewer does not think these claims are valid. - L.273-276: 'For the shallow layers, the distributions of the adapted models closely resemble those of the pre-trained models. However, as we move deeper into the layers, noticeable differences emerge. We attribute this disparity to the fact that dissimilarities between human hands and robot hands extend beyond low-level features like color and texture. Instead, they encompass higher-level features such as dynamics and structure.' At the very least, please cite relevant papers that analyze the adaptation between two domains. Especially, it would be very difficult to claim that 'they encompass higher-level features such as dynamics and structure.'. For what reasons did the authors draw this conclusion? - Fig.7: It is very hard to understand whether the keypoints are reasonable or not. For example, the bottom red star of the leftmost figure does not look meaningful. - Fig.6: Why do the authors specifically choose the three environments, specifically why only (c) is conducted in the 'Place Inside'? Since the differences between the proposed method and the others look minor for all three ablations, the readers can feel the environments can be cherry-picked. The reviewer thinks the authors need to show more results on different environments (can be in the appendix if space does not allow it).

Questions

- Fig.2: The data distribution changes between stages 2 and 3 are a bit confusing. The left human to robot should happen in stage 2 and the right one should happen in stage 3, correct? If it is, the review thinks it would be clearer to enclose the left one with blue dots and the right one with green dots. Otherwise, it looks like the adaptations are independent of the proposed three stages and are confusing. - Fig.2 and Sec.4: In stage 2, what do the dots between 'keypoint decoder' and 'reconstruct tgt view' mean? The reviewer read the implementation details shown in the appendix, but it does not explain the 'dots' part. - The ratio of parameter changes is written as 0.36\% in the abstract and conclusion but written as 0.18\% in the other parts. Which is true or the reviewer misunderstands something? - Fig.7: The reviewer cannot see the videos of how keypoints move in the video. - Sec.5.2: The reviewer is curious about the performance of the BC-only policy while adapting the domain gap using expert trajectories. Does it correspond to the performance at environment steps being zero in Figure 4?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

- As mentioned in the conclusion section, the method is tested on the same objects in the training set.

Summary

This paper studies a three stage framework for adapting pre-trained visual representations for dexterous manipulation tasks. Specifically, this work proposes only adapting a subset of network parameters in stages 2 and 3. The approach is tested on 12 manipulation tasks from Adroit and DexMV. These experiments demonstrate that the proposed techniques lead to more sample efficient learning. Furthermore, the overall approach outperforms alternative pre-training methods from recent work.

Strengths

- This paper demonstrates that using a visual backbone pretrained for (human) hand-pose estimation leads to improvements in learning downstream dexterous manipulation tasks over strong baselines from recent work. These results suggest that this inductive bias might be missing from recently proposed methods. - This work proposes adapting a specific subset of parameters during stages 2 and 3 of training, and demonstrates that this leads to more efficient learning. - The paper is clearly written and most of the details of the proposed approach are clear.

Weaknesses

Large parts of the proposed approach are directly applied from prior work including the pretrained visual representation (stage 1) and the learning objective from stage 2. This is clearly stated in the manuscript and appendix, and is not in itself an issue. However, this means the contributions of this work are primarily in the parameter efficient techniques used in stage 2 and 3. - Unfortunately, the results in Figure 6 indicate that stage 2 may improve sample efficiency, but may not improve asymptotic performance. Thus, it is unclear if this is a significant contribution. - Furthermore, the sample efficiency gains (Figure 6) are only demonstrated on two tasks (Hammer and Place Inside), so it is unclear how general these findings are. - Similarly, when compared to full finetuning (Figure 6c) it is not clear if asymptotic performance is improved and the sample efficiency gains are minimal.

Questions

- How does asymptotic performance compare in the ablations in Figure 6? - Do the sample efficiency improvements generalize to other tasks?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

Yes, the authors discuss limitations of the work.

Summary

This paper introduces a framework called H-InDex that uses human hand-inspired visual representation learning to solve complex manipulation tasks with reinforcement learning. The framework consists of three stages and outperforms other methods in challenging dexterous manipulation tasks.

Strengths

* A novel and effective method to perform dexterous hand manipulation using visual RL. * Extensive experiments and ablation study on various tasks. * A convincing framework exploiting the pre-trained vision models for robot learning.

Weaknesses

* The method seems only to be able to perform on a single object on each task, while the vision model should be able to generalize on category-level objects. * It is unclear of the RL action space, reward design, robotic control parameters, and the coordinate system used in the paper.

Questions

* Are the proprioceptive states of the robotic hand (e.g. joint states) considered in stage 3? * How does this work encode the object information in the compact representations?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed the limitations.

Reply by the reviewer

> QA1,2,3,6,7,8,9,10 The reviewer wants to thank the authors for the clarifications. Some other QAs still remain unclear. > QA4: "the quantity of keypoints is preset (we use n=30 in our paper), implying the potential for redundant or idle points." I think explicitly writing this in the paper would be better. Otherwise, readers (like me) will be confused. > QA5 " We acknowledge the insufficiency of the ablations and we want to respectfully emphasize that our ablation results are not cherry-picked; instead, the task is randomly sampled. During the rebuttal phase, we give a much more comprehensive evaluation of our Stage 2 and Stage 3. The results are given in the PDF file and clearly show the advantage of our method across tasks. We are also committed to releasing our codes." I don't think Fig.9 answers the original question while it still being very informative and comprehensive. Can the authors provide more results on ablations? Fig.9 is not related to Fig.6.

POST-REBUTTAL

I extend my gratitude to the authors for sharing their feedback. The additional results and analysis effectively alleviate my prior concerns regarding the technical details of the methodology.