Grasp as You Say: Language-guided Dexterous Grasp Generation

Summary

This paper proposes a novel robotic grasping task that utilizes natural language as guidance to generate grasping poses for dexterous hands that satisfy corresponding intentions. To accomplish this task, the paper first creates a large-scale language-guided dexterous grasping dataset. Moreover, the authors propose a progressive solution composed of a diffusion-based generative model and a regressive model to achieve generating intention-consistent, high diversity, and high-quality grasp poses for a wide range of objects.

Strengths

1. Compared to existing work, this work advances further by posing a more general and challenging problem and offering a more comprehensive solution. The introduction of this problem promotes the direction of general robotic grasping. 2. The paper contributes a grasping dataset that includes 18,000 objects with a total of 50,000 text-grasp pose pairs. This dataset fully utilizes the information provided by the Oakink dataset and transfers human hand poses to robotic hands in three steps. During the language annotation stage, the paper generates corresponding language descriptors by measuring the distance from the fingers to different regions of the object, then uses GPT to generate natural language. The entire process is technologically sound. 3. The paper adopts a progressive framework to address this issue, utilizing a diffusion-based generative model and a regressive model to ensure that the generated poses are both diverse and high-quality.

Weaknesses

Weaknesses 1. When verifying intention consistency, the paper uses chamfer distance and contact distance to compare the error between the estimated poses and the target poses. However, since the intention is derived from natural language, which is inherently imprecise and non-unique, using a collected target pose as the sole standard is inappropriate. For instance, for "to use a lotion pump, place your forefinger on the pump head and other fingers on the bottle body," there might be infinite ways to hold the lotion with the forefinger on the pump head, some of which might be different to the target one. Therefore, chamfer distance and contact distance may not be the best metrics. 2. The dataset collected in the paper includes only 50,000 grasping poses. Compared to existing datasets for parallel-jaw grippers, which contain millions or even billions of poses, the diversity is limited. For a dexterous hand, which has a higher degree of freedom, collecting more data is inevitable to achieve more varied grasping methods. I believe the performance on unseen categories would degrade significantly. 3. The authors mentioned that they use 20% of the data as a test set to evaluate the model's performance. However, they do not provide more specific information about the training and test sets. For example, whether the target poses and description texts of the same category of objects in the training and test sets are consistent, and what the results would be if given inconsistent descriptions, such as in Figure 6, "To grasp a wineglass, use your fingers to hold the stem securely" if changed to "To grasp a wineglass, use your fingers to hold the GLASSES securely" (I think this description will not exist in the dataset). 4. Although the authors use the Q1 metric to measure grasp stability, this analytical approach still has limitations. The authors could try using a simulator (e.g., Isaac Sim) to test the grasp success rate, which would be more indicative. Other Comments 1. The title of A.1.4 is "inference time," but the content is unrelated. 2. The real-world experiments in the video are not very promising. In the given demos, non-target areas of the objects are fixed by fixtures, leaving only the target areas available for grasping.

Questions

See weaknesses

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See weaknesses

Summary

This paper tackles language-guided dexterous grasping by creating a large-scale dataset pairing grasps and language guidance and presenting a grasp generation pipeline based on language instructions. The dataset, DexGYSNet, leverages HOIR and LLMs to scale up the size of grasps and annotations while keeping the generation costs manageable. The DexGYSGrasp framework introduces a two-stage progressive generation to balance diversity, quality, and grasp-language alignment. The method is evaluated on the intention consistency, grasp quality, and diversity against SOTA methods. Experiments demonstrate the necessity of progressive components, effectiveness of HOIR, potential to transfer to real-world scenarios. Overall, the paper advances the field of dexterous grasp generation by natural language guidance, providing a valuable dataset, and proposing a robust framework for generating high-quality robotic grasps based on human instructions.

Strengths

This problem of language-guided dexterous grasping is significant. This paper presents both a large-scale dataset and a method to generate diverse grasps. The dataset is a valuable resource for the community and the method can be integrated into a grasping policy and applied to real-world scenarios. The design of experiments justifies the choice to break the generation process into two stages and demonstrates the effectiveness of the method in improving grasp quality, diversity, and alignment with language instructions.

Weaknesses

1. The method requires a full point cloud and is designed for static pose generation. In real-world applications, access to a complete point cloud is often unavailable, although this could be mitigated by using a point cloud completion network. Moreover, grasping is typically a precursor to a subsequent action, requiring an end-to-end pipeline that can handle both grasping and the intended use of the object. An integrated approach would likely be more effective than segregating these steps into different modules. For instance, the subsequent policy must not only use the generated pose but also comprehend instructions (again, since the generated pose alone is not enough) and object geometry to perform the correct actions, which an end-to-end system could potentially handle more seamlessly. Plus, there can be slight adjustments to gestures after grasping a tool from a table but before using the tool. In this sense, real world experiments on this synthesis method are not so legitimate in the first place. 2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Given the diversity of real-world objects, it is crucial for the network to perform well on unseen items to be practically useful. Including more detailed experiments and analysis on in-category and out-of-category generalization would significantly strengthen the work. This could involve testing the method on a wider variety of objects and categories not present in the training dataset and assessing its adaptability given that OakInk has the category information.

Questions

1. The paper states that intention consistency is calculated by comparing the prediction to the most similar grasp target. While this approach addresses the issue of multiple valid grasp poses, could you provide more details on how the diversity of valid grasp poses is represented in your dataset and how it influences the evaluation metrics? 2. The paper does not seem to explicitly address how the proposed method generalizes to different objects, unseen objects, or even unseen categories. The dataset is derived from OakInk, which has category information. Could you provide more details on the performance and adaptability of your method when applied to objects and categories not present in the training dataset? This generalizability is important for the network to understand the correspondence between the geometry and functional semantics. In-category and out-of-category generalization are both essential for a capable network.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

1. The method is not yet ready for real-world applications due to several factors. It requires a full point cloud, which needs several adds-on modules like segmentation and completion for deployment. Additionally, there is a lack of discussion on how this method would benefit the subsequent grasping execution and how the grasped objects can be manipulated, which is crucial for practical use. 2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Ensuring that the method can handle a wide range of objects, including those not present in the training dataset, is essential for practical applicability.

Summary

The paper introduces "Dexterous Grasp as You Say" (DexGYS), a novel framework that integrates natural language instructions with robotic dexterous grasping. To support this integration, the authors present DexGYSNet, a dataset featuring 50,000 pairs of annotated dexterous grasps with corresponding human language instructions across 1,800 household objects, developed using innovative methods such as Hand-Object Interaction Retargeting and LLM-assisted annotation. Building on this dataset, the DexGYSGrasp framework is introduced, designed to generate robot grasps that are aligned with human intentions, ensuring diversity and quality. This two-phase framework initially learns a diverse grasp distribution aligned with intended commands and subsequently refines these grasps to enhance quality. Extensive testing both on DexGYSNet and in real environments confirms the framework's ability to produce practical, diverse, and high-quality grasps based on natural language commands, significantly advancing human-robot interaction capabilities for both industrial and domestic applications.

Strengths

1. The motivation of the authors is reasonable: leveraging current large language models to provide fine-grained descriptions of grasping states. The DexGYSNet dataset contributes to the community. 2. The authors validated their algorithm on a real robotic gripper, which is crucial. 3. The authors showcased parts of the DexGYSNet data in the supplementary material. Providing visualization code would further enhance this contribution.

Weaknesses

1. The evaluation metrics require further discussion. DexGYSGrasp is a generative framework, and the effectiveness of the proposed method based on Chamfer distance and Contact distance is not entirely convincing. Although the authors mention in Supp. (Line 521) that the target is "the most similar grasp", it is not always possible to find a closely matching grasp pose in the dataset. For instance, if the generated grasp is on the right temple of a pair of eyeglasses, but all dataset grasps are on the left temple, the Chamfer distance metric becomes ineffective. 2. DexGYSNet only annotates which fingers contact the object and which parts of the object are touched. It lacks more complex intents, which might limit its applicability. 3. The authors need to verify the quality of DexGYSNet. For example, what about the quality of the text annotations (e.g., could hallucinations from large models cause errors)? Additionally, the authors obtain dexterous hand parameters from MANO parameters through retargeting. They need to validate the grasp quality in physical simulations (e.g., Isaac Sim) to ensure that the objects are tightly grasped without dropping. 4. In Line 170, the authors train the Quality Grasp Component based on the most similar ground truth grasp with the same intentions. Perhaps, adding noise directly to the generated grasp pose to create a large amount of training data would be more effective? 5. The paper contains several typos. For example, Line 117 incorrectly uses $L_{spen}$. Lines 173 and 168 are repetitive.

Questions

My questions are primarily based on the Weaknesses: 1. The authors need to further discuss their evaluation metrics. *CD* and *Con.* are too stringent for a generative task. Metrics based on collisions or the standard deviation of poses seem insufficient for evaluating language-based generative tasks. 2. The authors might consider how to incorporate richer intents into DexGYSNet. 3. The authors should include an evaluation of the quality of the DexGYSNet dataset.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors discuss the societal impacts and limitations of their method.

Summary

This work proposes a language-guided dexterous grasp dataset based on OakInk, named DexGYSNet utilizing LLM and a corresponding two-component framework DexGYSGrasp which generates dexterous grasps based on human language instructions. Real-world experiments are conducted to validate DexGYSGrasp's performances.

Strengths

- The construction of the DexGYSNet dataset is cost-effective with the help of a large language model-assisted annotation system. The use of an LLM-based approach allows for efficient and scalable data labeling, which is crucial for developing robust models in this domain. - Extensive experiments are conducted to verify the performance advantages of the DexGYSGrasp method, particularly in terms of grasping intention and diversity, as well as the design of the evaluation metrics. The thorough experimental evaluation provides a comprehensive assessment of the approach and its capabilities.

Weaknesses

- While the DexGYSGrasp framework presents several notable contributions, such as the use of penetration loss and the Quality Grasp Component, the primary strength of the approach seems to be in the training methods rather than the overall framework design. - The Quality Grasp Component appears to function more as a post-processing step rather than a core component of the framework. This suggests that the key innovations of the work are centered around the training techniques employed, rather than the architectural design of the system.

Questions

- The language descriptions generated via the large language model, even with the inclusion of rich contact information as shown in Figure 6, still appear to lack precision. In this case, it is questionable whether directly applying metrics like Chamfer distance and contact distance is the most appropriate approach to reflect the consistency of the grasping intention. The suitability of these particular evaluation metrics for the language-based grasping task deserves further examination. - A minor question: the naming of the dataset as "DexGYSNet" seems somewhat confusing, as it suggests a naming convention more suited for a pipeline or system, rather than a dataset. "DexGYSGrasp" would be a more intuitive name for the dataset itself, as it aligns better with the specific task of grasping being addressed.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

See Weaknesses

Keep my original decision

The author's responses mostly address my concerns and questions. While some doubts remain on the foundational settings of this work, e.g., full point cloud and an end-to-end execution pipeline, this paper still provides a strong basis for future exploration in such topics. Therefore, I retain my original recommendation for acceptance.