Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects

Summary

This paper first proposes a new task, cross-category few-shot learning for articulated object manipulation. And it further proposes a framework that first explictly measures the semantic similarity of local geometries, interacts with the objects of novel categories according to the similarity, uses the interaction results to update the affordance estimation network, and finally manipulate the target object. Authors perform experiments in simulation and the proposed method outperforms existing methods. They show a successful demo on a real robot.

Strengths

The strengths of the paper are: 1. The paper formulates a new problem, cross-category few-shot affornace learning, which may be useful for robot applications; 2. The paper proposes a framework to address the proposed problem, which leverages the geometric similarity to guide the exploration on objects of novel categories. The paper designs a cross-category similarity learning pipeline to learn the similarity metric.

Weaknesses

1. Statistics, eg. std, are not provided in the experimental results, especially the variance casued by the choose of few-shot learning instances. 2. While the results of the proposed method are highed than baselines, the successful rate is still not statisfactory, especially the result of `full-data' is still low. More discussion on the results and illustration of failure cases can be added. 3. The assumption of the paper that similar geometry indicate similar affordance for novel categories may be not solid.

Questions

1. The `semantic' used in this paper is ambigious. You may add more explaination or use another term. 2. The similarity learning part needs more details. How do you use Eq(1) to learn the similairty? 3. How do you decide to stop the few-shot training? 4. What is sliding joint in L235?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The limiation of low success rate is not discurssed.

Summary

The paper focuses on the task of predicting affordances given a single-frame observation of an articulated object. The authors utilize few-shot learning to address the novel object adaption problem. However, previous approaches to few-shot learning require per-instance interaction during test time, which can be both costly and inefficient. To address this issue, this paper introduces the Where2Explore framework, which estimates geometric similarity across categories to offer more efficient interaction proposals. Additionally, it enables the transfer of learned affordance to novel cases. Experimental results demonstrate that this method surpasses previous approaches in various aspects.

Strengths

* The motivation to explore more efficient few-shot learning is intriguing. The concept of leveraging affordance similarities across categories is intuitive and seems to be effective based on the experimental results. * The training strategy to learn the cross-category geometry-aware affordance similarity is straightforward and effective based on the quantitative visualization. * This paper conducts extensive experiments to compare with prior work and includes sensible ablation studies to demonstrate the effectiveness of the Where2Explore framework.

Weaknesses

* There are not enough descriptions of the hard-coded intersection sets. Even if it’s proposed in the prior work, it’s better to include more details to be self-contained in this paper. This paper only chooses two actions, “push” and “pull” from the original 6 actions defined in the prior work. For the interaction definition, the action direction should be in the positive hemisphere of the surface normal. It’s not clear if it holds true for the real data. (Because the real data similarity visualization is not based on the surface normal) * In the few-shot adaption stage, it’s unclear how many interactions for each point are sampled for the similarity. In addition, the authors mention the similarity is conditioned on the action. It’s unclear if the action is still conditioned in the object/camera/world coordinate or some local coordinate to the interaction points. If the action is still parameterized in the global coordinate, it’s hard to see if the model can really learn the similarity. * For the few-shot learning, it’s hard to see the usefulness of the models in the supporting set. The current set of experiments cannot show the usefulness of the supporting set. Is this possible to learn from scratch with the few-shot learning strategy? * For the affordance and similarity training, there seems no reason to train similarity at the start, because the affordance predictions are still quite bad. It’s also not clear if training affordance and similarity in two-stage can achieve better performance.

Questions

* Why for the visualizations of the similarity prediction in figure 5, the action directions are different for synthetic data and the real data. Does the actual inference process have any difference for synthetic data and real data? For example, the sampled actions for few-shot learning are defined differently for the synthetic and the real data? * For the interaction budget, it seems that 1, 2, 5 get better and better performance, is there some performance top bar, and what’s the number of the budgets? * In the similarity visualization, why the dark blue is not considered as potential interactions (they should have the lowest similarity score?) Like the pot in figure 5 (the second row). It seems that the visualization mix the affordance and the similarity, which is a bit confusing.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors mention the limitation.

Summary

This paper studies the problem of interacting with novel articulated objects. More specifically, a few-shot learning based approach is proposed to adapt to unseen instances at test time. The methodology explicitly leverages similar local geometries, which may be shared across objects from distinct categories. Experimentation reveals superior performance relative to prior work.

Strengths

- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology. - Ablations are insightful. The ablations adequately test the utility of the design choices of the proposed methodology. - Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The fact that the proposed methodology achieves comparable performance to full-data while using only 0.3% of the data is impressive.

Weaknesses

- Simplistic problem setting. The problem setting seems to be simplistic. For instance, the robot arm is abstracted away. This completely side-steps the problem of finding a motion plan which actually solves the task in a collision-free manner, limiting real world deployment. Furthermore, the input point cloud is assumed to be cleanly segmented out. It is unclear whether this is a realistic assumption, especially for when the models are deployed to the real world. - Limited real world evaluation. It is not clear whether the model truly generalizes to the real world. Furthermore, the section on real world experimentation (both in the main paper and the supplementary) seems to be quite limited. For instance, how is the grasping (gripper open / close) done, as this doesn't seem to be an output of the system? Or, how is the motion plan obtained from the end-effector poses? More extensive (quantitative and qualitative) experimentation on real world data would be preferred to be convincing that the sim2real generalizes.

Questions

- Are "action directions / gripper orientations" often the normals? Is it possible to constrain the action space to just be normals? - Is a separate affordance model trained for each category, such that the similarity model can be trained on "held-out" categories? - Other alternatives / potential ablations: - As an alternative to the L1 loss for training the similarity module, could a cross-entropy loss have been used for whether the current affordance model was correct on the unseen object? This would also provide a notion of uncertainty of the model (via confidence), which could be used to guide exploration. - What if you returned the output of the affordance model for points with the highest similarity, without any further exploration? Wouldn't this correspond to points for which the model is the most confident in being correct? - Could you train an affordance model which only operates on smaller point clouds (ie smaller field of view, e.g. a patch)? This may force the model to only look at the local geometries which should be shared across categories? - "During few-shot learning, Where2Act will sample interaction on the points with an affordance score closest to 0.5." --> Why is this the case? Isn't 0 push and 1 pull? So shouldn't the highest confident action be sampled (ie farthest away from 0.5)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

There is a small section on limitations of the work, which could be expanded.

Summary

The paper focuses on few-shot affordance learning and articulated object manipulation. It introduces a framework named Where2Explore, which consists of the affordance module and the similarity module. Both modules share the same architecture, a Where2Act [24]-like network based on PointNet++ [28]. While the affordance module is trained on a single category, the similarity model is trained across categories to understand similarity in local geometry and manipulation semantics. The model is benchmarked on 942 instances from 14 (3 for train, 11 for test) object categories in PartNet-Mobility [37]. It outperforms previous works [24, 34] in simulation.

Strengths

1. The methodology is well motivated: Learning similar local geometry across object categories could benefit few-shot generalization in manipulation. 2. It is impressive to observe the stark differences between the seen categories and the unseen categories. 3. Under the same settings, Where2Explore exhibits a clear edge over previous works [24, 34] quantitatively. 4. The demo video is well-prepared.

Weaknesses

1. The method is limited to pulling and pushing actions, and the action trajectories are short-term and hard-coded. These limitations inherited from Where2Act could result in a significant simulation-to-real gap. 2. There is only a single example of real-world manipulation, which is insufficient to demonstrate the method’s capabilities. Besides, according to the video, it seems that the gripper may not always hold mugs firmly enough. More real-world experiments should be conducted, and their success rates should be reported. 3. Few improvements on the architecture. The current module largely follows the design of [24, 34]. While it is commendable to demonstrate new insights using a minimal network design, the current task may benefit from more fine-grained cross-category transfer learning. There is room for improvement in the methods. 4. Neural Descriptor Fields (Simeonov *et al*.) is also capable of capturing local geometry information using few shots. It would be valuable to discuss and even adapt them for comparison.

Questions

1. [38, 8] share a similar spirit and motivation in learning local geometry across categories and generalizing to the novel, which should be discussed in the related works. 2. Table 3: The successful rate of pulling drops for the full-data compared to ours. Any explanation for this observation? 3. Is the interaction budget (1, 2, 5) per instance or per category (10 instances)? This term seems not well defined. 4. L233: Which category is the affordance category ($C_{aff}$) in each combination?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper includes a section dedicated to limitations; however, it provides minimal discussion on the specific limitations. Please incorporate a comprehensive discussion of failure cases and limitations, including those in the Weaknesses section. To enhance the credibility of this work, it would be beneficial to incorporate quantitative results from real-world experiments and quantitative comparison to other works that follow a different paradigm, such as Neural Descriptor Fields and GAPartNet [8]. This would provide a more convincing evaluation of the proposed methodology.

Summary

The paper explores a few-shot learning strategy to predict the affordance of objects in novel categories. The key idea of the paper is to use a similarity module to transfer the knowledge from known categories to novel categories. Then the affordance prediction on the novel object can be learned with just a few exploration actions. The paper is compared with several existing methods on standard benchmarks.

Strengths

The paper addresses an important and interesting problem of transferring affordance knowledge from known categories to novel categories. The paper proposes a similarity module for effective affordance knowledge transfer and achieves quantitativel better results than existing methods. The paper is overall easy to follow.

Weaknesses

Method: I have a couple of questions regarding the learning procedure and intuition. 1. In Figure 2, to train the similarity module, the affordance module predicts the affordance map on the novel object and the similarity module predicts the per-point similarity. In my understanding, is the similarity module exactly the same with the affordance module but just learned with different data and learning objectives? If so, in eq (1), is the Aff the affordance module or similarity module in Figure 2? I am not sure if eq (1) and Figure 2 are aligned, maybe explaining in more detail could be helpful. 2. It is not clear to me how the affordance prediction is updated during exploration/inference. Is it more of a test-time-adaptation? What module parameters are updated? 3. I am not sure if the intuition of generalization using local geometries is well supported by the design. Is the similarity module responsible for this? If so, how do local geometries play a part? Isn’t it still predicted for all N observed points with global information? Since this builds the key idea of cross-category generalization, I think further clarification is necessary. 4. In the paper, 10 instances from each novel category are used for few-shot learning. I wonder, for example, for Where2Act baseline, is it also trained with only 10 instances? There is also no ablation on the number of instances required, could it be fewer? 5. The paper only explores push and pull. Maybe more actions can be studied to validate the method.

Questions

I think overall the paper addressed an interesting problem and proposed an effective solution, but I think further clarification on the method is necessary. Please see my comments above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.