SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation

Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could address your concerns: **Threshold distance choice for the contrastive loss.** Following the PointContrast paper, let $X_1, X_2$ be two partial point clouds from two views. We sample point pairs from them $(x_{1i}, x_{2i})$. We say that it is a positive pair if $||x_{1i} - x_{2i}||$ < 1cm and a negative pair elsewise. The positive and negative pairs are then plugged into the PointNCE loss defined in Equation 2. Here we note that, although this threshold is hard, the contrastive loss itself is a soft loss for two reasons: (a) the weighting in the soft PointNCE loss, and (b) the balanced sampling rate between positive and negative pairs. The specific threshold distance is decided based on the object scale and the overall framework is not too sensitive to this hyperparameter within a reasonable range. We will add more clarifications to the paper. **Is distance truly an effective criterion for distinguishing between similarity and dissimilarity?** For the feature similarities, we majorly follow prior works on their choices of metrics. Specifically: - We choose the cosine similarity for contrastive learning, as the InfoNCE loss with theoretical foundations (ref point contrast) - We choose L1-distance for computing the energy function during optimization, the same as in prior works such as NDF. Here the dominant factor for us to make the choice is not the criteria for similarity, but the optimization curve/landscape. We tried L1-distance, L2-distance, and cosine similarity here and observed that optimizing the hand pose with L1-distance gives the best quality and stability. **Symbols in Equation (3).** The # means the cardinality of the set defined in the brackets {...}, and the inequality $||f’_i - f’_j|| < \delta$ defines the region of the set. There are multiple notation systems for sets and we picked the one we’re used to. We apologize for the confusion and will add further clarifications to the paper. **Is the pruning process defined in Equation (3) considered reasonable?** The overall intuition of keeping 80% percent of points is that: (a) we want to discard the points with the worst features (here we recognize “bad” as lower feature consistency within local regions), but (b) we don’t want to discard way too many points to maintain the overall shape of the object/scene. **Could this pruning process potentially lead to the removal of critical edge information?** This is indeed a very good question. We think this should not be a big concern, as the pruning is majorly based on the features instead of point cloud geometry, and it is after the feature refinement. Sharp structures such as edges and corners usually easier for DINO to extract stable and coherent features. In fact, based on our observation, if we gradually decrease the percentage of points remaining (from 80% to 5%), the points leaving till the last are mostly the edge points which are more identifiable by the feature extractor. For 80% of points remaining, both edge and face points are kept and their distribution is fairly even across the object. We will add these discussions to the paper.

Thank you once again for your valuable comments on our work. We have already provided our response to your suggestions. As the deadline for discussion is approaching, we kindly request you to promptly provide any further questions or concerns you may have.

Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could address your concerns: **Sparse-view or depth-guided NeRF methods.** In our baseline comparison, we also used the depth for DFF (we apologize for not making it clear and will add clarifications to the paper). Therefore, we remove the difficulty of reconstructing the 3D geometry in both our method and the baselines and majorly focus on extracting the latent features for manipulation. Specifically, we want to address that, 3D consistency itself also exists in DFF/NeRF which directly averages the features from different views (because only the mean is considered and the variance is omitted). However, such view averaging with inconsistent 2D features can result in low-quality 3D features – which is the message we want to convey in this paper. **Fairer to utilize depth in the baseline.** - As mentioned above, for the DFF baseline, we are also using the scanned depth same as in our method for a fair comparison. We apologize for not clarifying this in the paper. - For UniDexGrasp++, we are using their state-based model which takes the ground-truth 3D geometry as input. We will add further clarifications to the paper.

Thank you once again for your valuable comments on our work. We have already provided our response to your suggestions. As the deadline for discussion is approaching, we kindly request you to promptly provide any further questions or concerns you may have.

Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could address your concerns: **Elucidating the advantages of distilling DINO features in the downstream tasks.** The features from large vision models such as DINO provide semantics-aware correspondences across different instances/categories, allowing one-shot/few-shot transfer of manipulation policies. However, the 2D image features are only sufficient for simple end-effectors such as parallel grippers operating on single points in the space. For dexterous hands with complex geometry, we need a 3D feature field to decide the parameters of all the finger joints, and that’s why we need such 2D-to-3D distillation. We will add more explanations and discussions to the paper. **The definition of one-shot should be explained in the manuscript (abstract or introduction).** Here the “one-shot” means that given one demonstration on the source object, we seamlessly transfer learned manipulations to novel scenes, accommodating variations in object poses, deformations, scene contexts, and even object categories. We abridge the wording a bit to avoid making the title too long. We will add more clarifications to the paper. **The input format and the dimension of the variables.** The 3D point cloud is obtained by merging the depth scans from the 4 views. We will add clarifications to the paper. **Motivation for using DINO and other possible choices of image-matching deep models like LOFTR.** Our framework is not specifically curated for DINO features and would be potentially compatible with any image feature extractors. We adopt DINOv2 here as they’ve shown strong cross-instance and cross-category correspondences. LOFTR seems to work mostly on matching two identical objects in different images. We will add the discussions of other image-matching frameworks like LOFTR to the paper. **Hyper-parameter for point pruning pruning. Multiple-stage or iterative pruning strategies.** We have tried different hyperparameters for the point pruning and within a certain range (70%-80% points left) it offers consistently good results. The overall intuition of choosing the hyperparameters is that: (a) we want to discard the points with the worst features, but (b) we don’t want to discard way too many points to maintain the overall shape of the object/scene. Muti-stage or iterative pruning would be an interesting future direction to study and thanks for pointing it to us. **How did the inference performance compare with baseline methods?** Our inference time is roughly the same as DFF, and both DFF and UniDexGrasp++ require iterative inference (RL or optimization) on the hand pose. This is because, for hand-like end-effectors with complex shapes, a variety of geometric and physical factors (e.g. contacts, penetrations, force closures, etc) need to be carefully decided for a successful manipulation (grasp). On the other hand, one-step forward prediction methods are prevalent for parallel grippers because they operate on single 3D points and can apply infinite forces. **Was depth information also utilized by DFF?** Yes, depth information is also used in DFF for a fair comparison. We apologize for not clarifying this in the paper. What we want to address in the paper is that, even with the scanned 3D geometry guiding the feature aggregation, the feature incoherency between different views, if not properly tackled, will lead to a lower-quality 3D feature field, causing a great drop in the success rate.

Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could address your concerns: **Novelty. It is natural that feature refinement induces 3D consistency.** Specifically, we want to address that, 3D consistency itself also exists in vanilla DFF/NeRF without any refinement that directly averages the features from different views (because only the mean is considered and the variance is omitted). However, such view averaging with inconsistent 2D features can result in low-quality 3D features – which is the message we want to convey in this paper. We agree that feature refinement w.r.t. 3D consistency gives better 3D consistency is not surprising. But what is not obvious is that feature refinement w.r.t. 3D consistency gives better cross-instance correspondences. Moreover, most contrastive feature learning (such as PointContrast) show their generalization after training on a large dataset of pairwise samples, but here we show that even only overfitted to the 4 views of the training scene with a few minutes of optimization, our refinement network naturally generalizes to novel scenes. Application-wise, compared to parallel grippers which operate on one single 3D point and can apply infinitely large force, dexterous hands require more accuracy in the end-effector pose to, contact the object surface, wrap the object, and get the forced closure. This makes a lot of methods for parallel grippers not directly applicable to dexterous hands. In such a scenario, fine-grained properties of the feature field become more important. **Noise in depth and camera pose.** All experiments for our method and the DFF baseline are on real robots, and the depth and camera poses are all from real data with scan errors and calibration errors. Only the UniDexGrasp++ baseline is run in the simulation environment with ground-truth 3D geometry (as we’re running their state-based model). The sim2real gap is a long-standing problem in robotics, and it is particularly severe for dexterous hands compared to simpler end-effectors such as parallel grippers. In this work, we show our results on real robots to demonstrate our robustness towards this sim2real gap. **Additional experimental comparison with NDF.** Here we provide additional comparisons with NDF. As we temporarily don’t have access to the hardware, we conduct the experiments in simulation environments (IsaacGym) with the object scans from the experiments in the paper. The success rates are: ———————————————————— Source: Mug2, Target: Mug2 (same object) - NDF: 8/9, Ours: 9/9 Source: Mug2, Target: BeerBarrel (same category) - NDF: 6/10, Ours: 9/10 Source: Mug2, Target: CatBowl (cross category) - NDF: 0/10, Ours: 7/10 ———————————————————— (*There are only 9 test samples for Mug2 because one Mug2 scan is used for providing the demonstration. We will update the results as well as conduct real-robot comparisons once we get access to the hardware again.) In addition, we also address the following limitations of NDF: NDF needs category-specific training and it takes a lot of time. NDF is specifically designed for category-level rigid object manipulation. It is not applicable to other scenarios such as deformable objects or cross-category generalizations. As a pure geometry-based method, NDF is highly sensitive to geometric variations (such as the BeerBarrel and CatBowl in the experiment here). Following the previous question on scan noises, our method actually shows much better noise stability than NDF in real-world experiments. **Other LLM/VLM backbones for feature extractions.** We choose the latest DINOv2 here as they demonstrated strong cross-instance/category correspondences on images, while some other VLMs are more focused on language queries (such as open-vocab query/segmentation). But we agree that exploring different VLM backbones, especially studying their view consistency and 3D awareness, would be an interesting future work. We will also add more discussions to the paper. **Other related works on learning joint hand-object poses or learning visual affordances from images.**. Thanks for providing these references. We will add the discussions to our paper.

Thank you once again for your valuable comments on our work. We have already provided our response to your suggestions. As the deadline for discussion is approaching, we kindly request you to promptly provide any further questions or concerns you may have.