Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions

Summary

This work formulates environment-aware affordance for articulated object manipulation, which incorporates both object-level per-point actionable priors and environment constraints. First, this work identifies limitations of previous works on object-level affordance learning, including over-simplified robot manipulators and environment occlusion; it proposes to incorporate robot and environment constraints into the per-point affordance score. Next, this work proposes a contrastive learning method to allow the learned affordance prediction model to generalize to complex and unseen occluder combinations. This work evaluates the success rate of the proposed method with open door and drawer tasks in simulated environments and show improvements over previous affordance learning methods, and it demonstrates that the learned affordance can be useful in real robotic tasks by showing real robot demos.

Strengths

- This work has a clear goal and motivation. It addresses several important limitations of previous affordance learning works (over-simplified robots and environments), allowing per-point affordance prediction to be deployed on real robots to complete tasks in more complex and realistic environments. - This work provides adequate quantitative and qualitative experiments and comparison with baselines to show the effectiveness of the proposed method. It also justifies the necessity of 2 significant components, occlusion field and contrastive learning, with ablation studies.

Weaknesses

1. This paper does not provide a clear definition of per-point affordance in problem formulation. I think this paper needs to provide a more comprehensive definition of "affordance" and "interactable" in introduction or section 3, preferably with concrete examples. Currently the motivation and goal of this paper is hard to understand without reading a previous work such as Where2Act. 2. The problem formulation is also missing several key points. - What is the scene point cloud? Based on the experiment section, it seems to be a colored point cloud obtained from a depth camera. This information needs be explained clearly in the formulation. - How is the point cloud segmented into target point cloud and occluder point cloud? This information is available in the simulation, but how is it done in the real-world experiments? 3. The robot-target conditioned occlusion field is not described clearly. I find section 4.3.1 very confusing. The occlusion field is defined on a open and connected set $S$, but I cannot see how $S$ is constructed. I assume that $S$ is the space left after *filtering the unimportant points whose field values are too large*, but the field is constructed with cross-products, and it is not clear which norm is used to define *large*. Moreover, $S$ clash with the notation of the scene point cloud (which is clearly not open and not connected). I believe this section needs to be reworked to provide a clear explanation.

Questions

1. How are the simulation assets pre-processed? What convex decomposition is used? - With default PartNet-Mobility and SAPIEN simulator, all assets would be processed into convex shapes and form a large gap between physics and rendering. Manipulation experiments performed on such data would not reflect a realistic scenario. So it is important to describe the data pre-processing method. 2. Why robot position $R$ is used as input instead of transforming the point cloud? Is there some important choice on canonical frame that is not described in the text? How do you ensure the model prediction is invariant to a rigid transformation of the entire scene? 3. What are the setup for the real-world experiments? How are the models trained and deployed? Is there finetuning on real-world data? Is there domain gaps when the model is trained on perfect point clouds in simulation and deployed on noisy data in the real world? 4. What is the inference time for the method when deployed on real scenes?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper provides a description of some limitations and I have raised questions of other potential limitations in the previous *Questions* section.

Summary

The paper introduces the environment-aware affordance task, taking into account both object-centric affordance and environmental constraints. More specifically, the proposed task is designed to incorporate various occlusion cases and the robot's position. In order to tackle the inherent challenge of combinatorial explosion, the paper introduces a novel framework that utilizes a local robot-target conditioned occlusion field and contrastive losses. This framework enables effective training on simpler scenarios with only one occluder, while also facilitating generalization to more complex scenes.

Strengths

* The proposed task that takes the environment constraints into account is novel and insightful. Occlusion and robot position conditioned affordance is intuitive and potentially useful for downstream robot tasks. * The idea of utilizing a local robot-target occlusion field to enable generalization to more complex occlusion scenarios is straightforward and effective based on the experimental results. * The strategy employed to construct positive and negative pairs for the contrastive loss is intriguing, and the ablation studies confirm its efficacy in improving the scene encoder.

Weaknesses

* In the proposed environment-aware affordance task, both the segmentation of the target object and the occluded objects are provided. For previous work like W2A, which only consider the object-centric affordance, it’s unclear whether the addition of simple heuristics can yield positive results. One possible heuristic is to examine the predicted affordance from W2A and determine if there are any points close to the points of occluder, subsequently filtering the affordance based on this observation. While I acknowledge the significance of environment-aware affordance for downstream tasks, I remain uncertain about the complexity and realism of the current proposed task. In addition, for the target objects, there are only cabinets and tables, it’s also unclear if the train and test scenario uses the same instance or different instances from the same category. Only two categories may not be challenging enough for the task. * For the robot position, it’s unclear how many positions are chosen in the train and test set. From the supplement video, there seems to be 4 different robot positions. It’s unclear if the method can generalize to new robot positions well. * For the push and pull action, it’s unclear if the default action direction is the surface normal. There should be more descriptions on the action selection and the dataset construction, including how to pick the occluder, how to add the occluder and how to judge if the action succeed considering the occluded objects.

Questions

* What’s the performance of the trained model on scenes without any occlusion? And what will be the performance compared to the prior work in this scenario? * When inference, is the segmentation of the target object and all occluded objects are given? For the real case, how do you get the segmentation? * Is the number of points of the target object fixed for the network? How about the number of points for the occluded objects? If the number of points is not fixed, how to balance the number of points in each train example?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors mention the limitation.

Summary

Given a 3D point cloud and a robot, the method predicts per-point actionable affordance. More specifically it predicts whether a given point is pushable or pullable by the robot in a given position. The method takes into account environment occlusion. The paper presents a data-efficient training framework that allows training on a single occlusion and generalize to multiple occlusions. In addition, the paper presents a novel contrastive learning approach.

Strengths

- Acounting for occlusion when predicting affordance is a step up from previous work. It has not been tackled before as far as I know. - The paper presents several novelties: problem setup and formulation, data-efficient framework, and contrastive learning approach - The method outperforms previous work. - Paper is well written and easy to follow. - The results are well presented and accompanied by a good analysis

Weaknesses

- The action space is too small. Only pushing and pulling are considered. - The test setting seems too simple. Most of the results are shown on very similar desks. - Related work section is too short and does not clearly explain what is the gap in previous work, nor how the paper is filling that gap. - Section 4.3.1 is lacking a supporting figure - The robot geometry is not taken into account.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The paper presents a novel environment-aware affordance prediction framework for robot interactions (pushing/pulling) with parts of articulated objects (drawers and doors). The proposed approach consists primarily of two novel improvements: a robot-target conditioned occlusion field and a contrastive learning formulation for improved data efficiency. The paper evaluates the proposed approach against where2act and O2O with metrics from those works and demonstrates a fair improvement (~6% overall and 2-10% in all individual categories). Real world trials demonstrate the usefulness of predicted point-level affordances for robotics tasks. Qualitative results demonstrate several clear wins for the approach related to prediction of occluding object constraints.

Strengths

Based on a known simulation system and asset data, the evaluation and metrics soundly demonstrate the advantage of the proposed approach over prior art. Trials on down-stream manipulation tasks with a real robot platform validate the usefulness of the point-level affordances predicted by the approach. The proposed improvements over prior art (contrastive learning and robot-target conditioned occlusion fields) appear novel and constitute interesting solutions to the challenges posed. The advantage of training on simple occlusions and generalizing to more complex occlusion scenarios is a clear strength of the approach.

Weaknesses

Some areas of the paper could be improved. I felt there was some information missing leaving me with questions. The high-level concepts were repeated often and could be deduplicated to make room for more details. The line was slightly blurry between novel work and prior art. Clearly this approach builds upon W2A and O2O, but the paper would benefit from clarifying carry-over vs. novel formulations. SMA metric appears to be the ssr metric from W2A with a different name. Some discussion of F-Score and Average Precision metrics would help interpretation. Could be appendix if repeated from prior art. Overall accuracy improvement over SoTA was somewhat small (6% over prior methods to achieve 43/36 push/pull). The paper talks up the approach without mentioning remaining challenges or failure modes. Exposition could be improved with further polish and proofreading. Grammar(G)/Typos(T) (listed with line number) : G - 18 - perceiving and manipulating these objects present T - 39 - also faces experiences T - 107 - objectc T - 148 - extra period after fR G - 255 - We randomly adopt proposed T - 267 - Moreover to we 274 - Table 3 2 (formatting? Comma or conjunction?) G - 284 - of that may exist collision

Questions

I understand the value of robot and target encoder networks, but I didn’t find any satisfying details or citations to provide insight into what these networks produce/encode or how they are trained. Are these empty MLPs initialized with random weights and allowed to fill freely, or is there something else to them? Are they described more in prior art or specific to this work? I would like to better understand the intuition behind the Robot-Target Conditioned Occlusion Field. Why does the vector field of cross products of point-to-robot and point-to-target vectors provide a good heuristic? Some expanded text here would be valuable. What is a “novel” scene for the evaluation test metrics? Is it a scene with additional occluders that was not present in training? Why does the pulling performance on novel scenes only improve 1-2%? Is this significantly harder to predict occlusions or is there a common failure mode that escapes the assumptions of the heuristic for the approach?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Discussion of limitations seemed light. Only the lack of mobile base was mentioned. Given the maximum performance of 43/36% push/pull I expected to learn more about what wasn’t working or what additional challenges and failure modes were observed.

Summary

This paper introduces a framework for learning environment-aware affordance in occluded 3D object manipulation. It tackles combinatorial explosion and improves data efficiency. The authors showcase their superior framework using benchmarking multi-object full-robot environments in the SAPIEN simulator. Their contributions involve studying articulated object manipulation within environment constraints, proposing a data-efficient framework, and establishing benchmarking environments.

Strengths

- The framework proposed addresses the learning of environment-aware affordances for 3D articulated object manipulation, even under occlusions. - The paper focuses on affordance learning and introduces benchmarking environments to enhance robotic system performance in real-world settings. - The paper includes a comprehensive description of the framework, establishes benchmarking environments, and highlights its superior performance. - It is well-structured, easy to comprehend, and effectively presents experimental results.

Weaknesses

- The proposed framework has only been evaluated on simulated environments, and there is a lack of real-world experiments from different perspectives and scenarios. - This work primarily examines a static scene, which raises concerns about the applicability of the proposed framework in real-world scenarios that involve more complex and dynamic environments. - The authors have not provided a detailed analysis of the failure cases of the proposed framework.

Questions

- L38/267 mention that sampling-based motion planning faces performance degradation with increasing occluders. Could you provide a benchmark of the time and cost of different methods to address this issue? - L147 mention E_R encodes the robot position R into a latent representation. What about the geometry or configuration of the robot? How do you ensure that the scene and the configuration of the robot are plausible? - How would the proposed framework generalize to different types of robots and scenes? - In real-time executions, the state of point clouds is constantly changing, and the heatmap also changes. How do you plan to handle this issue?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations and potential negative societal impact of their work.