CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects

Response to Reviewer UC4D

Thank you for your thoughtful comments and suggestions. We addressed your comments, and would like to provide additional details regarding your questions as follows: > (W1) The scope of manipulation tasks in this research is restricted to single-object state maneuvering using a closed gripper. Incorporating more intricate robot-object interactions, such as grasping, could help to fully utilize the robot's kinematic capabilities and cover more complex scenarios. While we agree that our scope involves manipulating a single object using a closed gripper, as far as we know, this is the first time that anyone has gotten a real robot to do non-prehensile manipulation that utilizes full 6D end-effector pose on the diversity of objects that we had. Most of the existing methods are limited to either simple primitives (e.g. poking objects) [1] or are limited to objects for which they know the physical parameters (mass, friction coefficients, center of mass) [2]. Also, grasping using a simple gripper ***does not*** involve as intricate robot-object interactions as non-prehensile manipulation — it simply involves reaching the object, and then closing the gripper, and has been studied significantly in the past [3]. In contrast, because non-prehensile manipulation requires careful choice of contacts, mode switches, and joint movements to maneuver the object, it is known to be a more difficult problem [4]. > (W2-1) The paper measures the success of the tasks using a "success rate." However, necessary details, like the specific criteria used to determine successful manipulation, lacks clarity. > > (Q1) Could you elaborate on the "success rate" metric used in your experiments? What specific criteria are used to determine a successful manipulation? We describe task success criteria in section 4.1 as the translational offset of the pose within 0.05m distance and angular 0.1 radians. We updated the draft to better elucidate this point in the experiment section. > (W2-2) Moreover, the goal state illustrated in the manipulation videos (or their screenshots) appears to be a snapshot of a future state, which does not accurately represent the actual desired outcome. The goal visualized in the video ***is*** the approximation of the object at the actual desired pose. Obtaining goal images of objects after positioning objects precisely at the goal pose is impractical; we believe this is a reasonable visualization. Below is the actual difference in average pose error between the final object pose and its desired goal location in the real-world experiments. | Metric | Value | | --- | --- | | Translation Error (m) | 0.036 ± 0.010 | | Orientation Error (rad) | 0.068 ± 0.019 | > (Q2) Can you explain how do you measure the physical parameters in the real world and make sure they are aligned or well-simulated in simulations? Actually, we don’t need to measure the physical parameters. The purpose of distillation (section 3.1) is to remove the need for privileged information, such as the physical parameters. We distill our teacher policy (section 3.1) so that the student policy can operate in the real world without the physical parameters as input. The student architecture, detailed in A.1, also shows that it does not require physical parameters as input. [1] Zhou, Wenxuan, et al. "HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation." *7th Annual Conference on Robot Learning (CoRL)*. 2023. [2] Cheng, Xianyi, et al. "Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d." *2022 International Conference on Robotics and Automation (ICRA)*. IEEE, 2022 [3] Sundermeyer, Martin, et al. "Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes." *2021 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2021. [4] Lynch, Kevin Michael. *Nonprehensile robotic manipulation: Controllability and planning*. Carnegie Mellon University, 1996.

Response to Reviewer dbmK

We appreciate the reviewer’s effort in providing the review. Unfortunately, the reviewer seems to have a major misunderstanding about our paper and raised concerns based on this misunderstanding. First, our goal is not to learn a “collision-free policy” that computes a collision-free motion. As the title of our paper and Figure 1 on page 2 suggest, our problem is that of non-prehensile manipulation, and object manipulation cannot be done without making contact with an object (We suggest the reviewer to checkout the video of our robot performing the task of interest: [https://sites.google.com/view/contact-non-prehensile](https://sites.google.com/view/contact-non-prehensile)). In particular, our problem involves learning where to contact the object and how to apply forces to an object, which involves deliberately colliding with the object. The key aspect of this problem, therefore, is knowing where the robot has contacted the object because that tells us what forces the robot can apply to the object. Our contribution is the design of pretext task and neural network architecture tailored to detect such contacts. The reviewer raised two concerns based on this misunderstanding. The first was the naming of our method (it has “contact-based” in it). We hope the reasoning above tells you why contact-based representation is important. The second is whether our contact predictor generalizes to novel objects, because even a small mistake would make the robot collide with the object. We hope the reviewer now has a better understanding of the task, and how this would not be an issue. > (W1) The novelty of this work is marginal. basically, it just applies point-cloud-based reinforcement learning to non-prehensile manipulation tasks. - While there have been approaches that consider point-cloud-based RL in robot manipulation, no prior work has solved closed-loop non-prehensile manipulation of general objects. The inherent complexity in contact dynamics between the object, robot and the environment in non-prehensile manipulation renders naive RL application to be prohibitively expensive in our domain without nontrivial design choices, as seen in our results (Figure 7). As such, no prior work has demonstrated the generalization over diverse object geometries with closed-loop feedback control in this domain. > (W2) The generation of the collision label is questionable with the coverage of the collision states, i.e., if this work truly aims for generalization on unseen objects, I wonder how the collision prediction network be generalized to the unseen geometry. Besides, if the contact network is trained as a guide for the encoder, how does it guarantee to generate a collision-free policy? The collision decoder itself is not perfect (due to the coverage issue and neural network prediction), and the influence on the encoder is indirect (leads to less perfect), not to mention the policy is distilled from the teacher network to the student network (even less perfect). > Can collision network generalize to unseen object geometry? How can we ensure that the policy can precisely avoid collision despite errors coming from factors like (indirect guidance, limited coverage and approximation error from distillation)? - Yes. Our network generalizes to unseen geometry, as evident in our experimental results. All except one of real-world objects are unseen, and the simulation test setting involves novel objects as well. We kindly encourage the reviewer to give a read of our experiment section. > (W3) Then, though the authors have conducted extensive baseline methods on the encoder side, in my opinion, they should also discuss different decoder schemes to justify the "contact-based" name. For example: > > (a). the encoder is directly trained by the policy network? That is, no contact decoder to pretrain the encoder. > > (b). the decoder is a geometry reconstruction network. In this sense, the encoder can capture the geometry information of the object point cloud and hand. I see no apparent reason why the "action and rewards" cannot be done with such kinds of decoders. - The baselines you mentioned have already been implemented in the experiment section. - In Section 4.2, the End-to-end (E2E) baseline is the one that the encoder is directly trained by the policy network i.e. RL objective. This corresponds to option (a) from your recommendation. - In Section 4.2, the Point2Vec (P2V) baseline represents the pretraining scheme where the encoder is trained on shape-reconstruction objectives. Point2Vec learns shape representation by reconstructing the latent shape embeddings (similar to data2vec[1]). This corresponds to option (b) from your recommendation. - Ours outperform these both in terms of data and computational efficiency. (Figure 7) [1] Baevski, Alexei, et al. "Data2vec: A general framework for self-supervised learning in speech, vision and language." *International Conference on Machine Learning*. PMLR, 2022

Response to Reviewer dbmK (1/3)

> Let me explain why I think the contact-based is not proper, or at least not clear. > > First, contact is a kind of state when the robot gripper/hand touches the surface of the objects. It is not a state where the gripper is away from the object's surface or inside the object's surface. We can distinguish these states with the term "collision". "contact" is surely in "collision", but "collision" is not "contact". "contact" is an intermediate state between "in-collision" and "collision-free". In my view, the author misuses these two terms. I suggest the author refer to Springer Handbook of Robotics, Contact Modeling and Manipulation, and check related sections. ([https://link.springer.com/referenceworkentry/10.1007/978-3-540-30301-5_28](https://link.springer.com/referenceworkentry/10.1007/978-3-540-30301-5_28)) > > In the domain of robotics, the term "contact" has a special meaning. It refers to the contact interface. If the authors only use the word "contact" as a normal verb, as they did at the end of Sec. 3.2, "A patch is labeled as 'collision' if any point within it contacts the gripper", and use contact/collision interchangeably, I can accept that, but please make it clear. > > I have a feeling it is the same reason why reviewer LafH thinks this term usage is problematic. But I will only speak for myself. * The main point raised by the reviewer here is to disambiguate the use of “contact” (for signed distance d == 0) and “in-collision” (d≤0). We agree that this disambiguation is important in collision-free motion planning, where the penetration depth is used to define the cost function for trajectory optimization, for instance. We use the two terms interchangeably in our setup, since there’s little point in discriminating the two conditions in our task. * As we train an RL agent, the training process occurs primarily through robot-object interaction. In real-world interactions, penetration between objects cannot occur; and in a simulation, the momentary penetrations are handled by the physics engine itself during contact resolution. Thus, from the perspective of the agent, there’s no need to disambiguate the two conditions (d==0, d≤0). * That being said, we appreciate the reviewer’s suggestions on clarity. We have updated Section 3.2 to communicate that the pre-training labels primarily consider penetration between objects. > Another way to interpret this is that the contact surface is indirectly determined by in-collision/collision-free, just like how SDF indirectly defines the surface by inside/outside indicators. In this sense, it will raise the issue of generalization to unseen geometry, which happens to SDF-based 3D reconstruction tasks. Of course, I noticed the claim of generalization to unseen geometry in the main paper, but the objects used are quite similar to trained objects. I just don't think it will generalize to a larger object set (it already does not perform well on slightly complex objects, Table 1). * We appreciate the reviewer’s concern that SDF-based 3D reconstruction tasks often suffer from generalization. Of course, it’s only natural that representations learned through neural nets would be bound to generalize within the distribution of objects similar to the training set. However, we’d like to clarify that our objective is to predict the collision between *local* patches and the robot gripper, rather than the much harder task of predicting the *global* implicit shape. This type of *locality* is often utilized in prior works[1,2] as a key ingredient to generalizing across diverse shapes, as the geometry of local parts tends to be quite commonly shared across different objects in a combinatorial manner, thus boosting the generalization over different shapes. * While the reviewer wonders whether the performance drop on “complex” objects is due to the difficulty of generalizing to objects of complex shapes — which could be partly true — we believe that this has more to do with the severe occlusion and dynamic instability of objects rather than the difficulty of our representation to generalize to complex geometry, as discussed in Limitations (Section 4.2).

Response to Reviewer LafH (1)

Thank you for your thoughtful comments and suggestions. We addressed your comments, and would like to provide additional details regarding your questions as follows: > (W1) Lack of Unique Design for Complex Operations: The paper emphasizes in the introduction that its approach can execute more complex robotic arm operations than prior grasping work. However, in the method description, there appears to be no clear unique design to directly support this motivation, raising doubts about the novelty and effectiveness of the method. - We’d like to highlight two unique design choices made to address the complexity of the non-prehensile manipulation task: the model architecture and object representation learning. - The complexity of contact dynamics of non-prehensile manipulation necessitates closed-loop feedback control. However, training such a dense, closed-loop policy is expensive in terms of both time and compute cost. To address this, we first adopt a patch-based transformer architecture, for the first time among works in point-cloud-based robot manipulation. This accelerates the training process and reduces the training time remarkably, as shown in Figure 7. - Secondly, we observe that non-prehensile manipulation is highly sensitive to the knowledge of contact between the object and the robot. To reflect this, we adopt a novel contact-based pretraining task. Together with our model architecture, we demonstrate significant efficiency gains as a result of adopting our pretraining scheme. > (W2) Overemphasis on "Contact": While the authors place particular emphasis on the importance of "contact" in nonprehensile operations, it seems that the equally vital role of "contact" in grasping tasks has not been adequately considered. This imbalanced emphasis may impact the comprehensiveness of the method and its practical applicability. > > (Q1) The paper strongly emphasizes the role of "contact" in nonprehensile manipulation. How does this emphasis differ in importance from its role in grasping tasks? - Non-prehensile manipulation requires careful choice of contacts, i.e. sequence of contact mode switches which defines what forces the robot can apply to the object. We strongly encourage the reviewer to check out the video of our method operating in the real world[[https://sites.google.com/view/contact-non-prehensile](https://sites.google.com/view/contact-non-prehensile/)]: this is in stark contrast with grasping with a parallel jaw gripper, which typically only involves approaching an object and closing the gripper. > (W3) Questioning the Novelty of Point Cloud Processing: The paper utilizes a Transformer-based architecture to process point cloud data, but this approach does not appear groundbreaking, emphasizing the need for a more detailed description of the policy network design. - Most prior works in robot manipulation actually only use PointNet-based architectures. We explore the choice of point cloud processing architecture, and show that patch-based transformer greatly improves computational efficiency in our domain, by about 8x in terms of model inference time and more than 20x in terms of policy training. To our knowledge, we are the first ones to demonstrate the efficacy of patch-based transformer in point-cloud based robot manipulation. > (W4) Insufficient Description of the Policy Network: The paper provides a relatively concise description of the policy network, lacking in-depth details, which might hinder readers from fully understanding the workings of the method and its potential advantages. > > (Q2) The description of the policy network is relatively concise. Could the authors provide more details on its design and working principles? - We apologize for missing the details, and updated the paper to include additional details. The policy starts with an initial cross-attention layer against the tokens of the current task context(joint state, previous action, goal, and physics parameters) to extract task-relevant information from the patch-level embeddings. The resulting features are concatenated again with the task context, which are then processed by an MLP(multi-layer perception) before being fed to actor and critic networks. Given that our focus is on the point-cloud processing representation model, we placed less emphasis on the policy network architecture when describing our model. We have now added extra details about the policy network to facilitate the reader’s understanding of our model. (Figure 5. & Section 3.1)

Response to Reviewer 4HBK

We thank the reviewer for the effort and positive assessment of our work. > (W1) The model takes the difference between current object pose and target object pose as input. This can bring significant engineering challenge since object segmentation and pose tracking in the real world can be difficult. However, the authors documented their approach very well in Sec. A.3. > > (W2) The method only works on object well separated from clutter on a tabletop. This is related to the above assumption for the object pose, since having more than one object in close contact will make object segmentation and pose tracking even more challenging. We agree that segmentation and pose tracking are significant engineering challenges, and we appreciate the reviewer for recognizing the documentation of our approach. We are also hopeful that recent advances in foundational vision models, such as SAM[1] or POPE[2] can facilitate object pose tracking and segmentation to be more generally accessible for robot manipulation in the future, particularly for the challenges such as segmenting non-singulated objects. > (Q1) What is the coordinate frame of the hand pose input? Since the hand pose is sampled near the object to ensure good data balance for contact/no contact during pre-training, will the hand pose go out of distribution during policy learning? The hand pose is represented in the world frame. Since the poses are sampled from the workspace during pre-training, it does not actually go out of distribution during policy learning. > (Q2) In Figure 7, is the scale of the success rate the same for the two plots? It seems that the final success rate doesn’t match up for the green, red, purple and brown curves. There was a small bug in the plotting code, which has been now addressed. We updated Figure 7 and made minor adjustments on reported metrics in Section 4.2. Thank you for catching that! > (Q3) Which simulator is used? I didn’t find it in the paper. We used Isaac Gym. We added the reference to the simulation in Section 3.1. [1] Kirillov, Alexander, et al. "Segment anything.”, ****Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4015-4026 [2] Fan, Zhiwen, et al. "POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference." *arXiv preprint arXiv:2305.15727* (2023).