Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image

Summary

The paper presents a novel task focused on the reconstruction of multiple articulated objects, considering part-level shape, pose, joint parameters, and part-instance association, using only a single RGBD image. The authors propose an effective detect-and-group strategy that harnesses the part-level representations to detect, reconstruct, and predict parameters for articulated objects with diverse structures. Additionally, to enhance the detection performance, the paper introduces an oversampling and fusion strategy during inference. Moreover, the authors incorporate anisotropic size normalization and a refinement module to improve reconstruction quality and enhance part pose/motion prediction.

Strengths

* The proposed task of reconstructing any number of articulated objects is both novel and valuable for exploring potential downstream robotics applications. * Utilizing part-level representations and employing the detect-then-group strategy are intuitive and effective when handling articulated objects with diverse structures. * The proposed end-to-end method builds upon 3DETR by incorporating part-level detection and pose/motion prediction. Furthermore, the authors introduce an instance-loss to aid in grouping within the part latent space. * The experimental settings are reasonable and allow for meaningful comparisons with previous work in the fields of articulated object reconstruction and motion prediction. The results demonstrate the effectiveness of the proposed method.

Weaknesses

* Due to the task's focus on reconstructing part shapes, it’s crucial to show some novel viewpoint of the recons trued shapes. For example, the reconstructed shapes of some base parts are hard to see the improvement only with quantitative results. Additionally, all visualizations in the paper appear to align with the input RGBD viewpoint, limiting the comprehensive understanding of the reconstructed shapes. * If I understand correctly, for the statistics in table 2, the instance number includes the same articulated objects with various part states, or it’s hard to explain why there are so many instances in each category (much more than the object number in SAPIEN dataset) * For the evaluation, it seems that in the test set, the part state are randomly initialized. Then most doors or drawers are actually open. However, in the real case, most time, the movable parts are closed. It’s better to have separate evaluation for the models with different motion states * The paper lacks statistics on the number of articulated objects present in the input image, and it would be intriguing to include evaluation results that consider the number of objects to better understand the performance of part-instance association. * The quantitative improvements achieved with QO, PF, and kIoU metrics do not appear to be substantial. Are there any qualitative results available to provide additional insights or demonstrate the effectiveness of the proposed module?

Questions

* Why the number of images in the validation set is much more than the number of images in the train set and test set? * For the number of objects in each image, what’s the statistics in the train set and test set? * For the model, why “ours-BG” sometimes outperform “ours”? “Ours” should have extra information of the foreground mask, right? Is this caused by the imperfect performance of the foreground mask?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors mention the limitation in the supplement.

Summary

This paper presents a new method for 3D semantic instance reconstruction at the part level from a single RGB-D image. This method follows a top-down manner by first detecting object parts using 3DETR, where each instance part's bounding box will be predicted. The point cloud located inside each bounding box contains the part surface geometry, which will be normalized into a uniformly-scaled canonical system for the following part shape reconstruction. The key contribution of this paper is the kinematics-aware part fusion module, which follows a bottom-up manner to constitute complete instance shapes from parts. The paper writing could be improved, e.g., the first paragraph in the introduction section is actually elaborating on related work. An illustration in Sec 3.6 would be more accessible for the audience to follow. The experiment metrics and designs are extensive but lack some comparisons with the state-of-the-art.

Strengths

In my view, the major strengths can be concluded as follows: 1. Kinematics-aware part fusion (KPF). The authors follow a top-down and bottom-up manner to first detect part-level geometries from the point cloud. Then they use KPF to construct instance shapes from parts. These two processes are trained end-to-end jointly to improve the entire performance. 2. It shows good qualitative and quantitative performance in both synthetic data and synthetic data, with an acceptable generalization ability, even though they only train their model with synthetic data.

Weaknesses

The weaknesses of this paper are also obvious. 1. This paper follows a similar top-down pipeline as previous works. (e.g., RfD-Net). In L34, the authors argued that previous methods "are either limited to a single instance or require a separate instance detector, making them not end-to-end trainable". I do not think this makes sense, as this paper also requires an instance detector. Besides, relying on a separate instance detector does not mean they can not be trained end-to-end. 2. As this task is not new and this paper follows a similar pipeline, it means the contribution of this paper only comes from the module design. To me the major contribution lies on KPF module. But from the ablation study, it seems KPF does not contribute the most. 3. Section 4.2, comparing with OPD by adjusting the IoU threshold is not rigorous. There is no theoretical guarantee that this comparison is fair. Since OPD is an RGB-based method while this paper uses RGB-D information, a fair comparison would be to add depth information into OPD or to remove the depth channel in your method.

Questions

Here are some suggestions to improve this work, with which I would consider raising my score. 1. A comprehensive analysis and comparison between the key module of this method (e.g., KPF) and the state-of-the-art (A-SDF is a bit old). 2. The introduction should be about telling the story of the paper's motivation, key observations and contributions. Listing many related works there made it difficult to follow.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This paper well discussed the limitations and failure cases in the supplemental.

Summary

The paper presents a detection-based reconstruction method for articulated objects along with estimating part-level 6D object poses, sizes, and joint parameters. The paper uses 3DETR as the backbone predicts all of the above quantities while treating this problem as a supervised learning approach given labeled synthetic data. The approach trains only on synthetic data and transfers to real world dataset.

Strengths

The paper effectively uses a detection backbone i.e. 3DETR to reconstruct multiple articulated objects from a single view RGB-D observation. Sim-to-real transfer uses only synthetic data pretraining is a strong result. The use of set matching (from instance segmentation) literature for articulated part reconstruction is interesting. Joint estimation and shape reconstruction quantitative results are significantly better than baselines. I believe breaking the problem in this way i.e. part level reconstruction omits the need for model selection (i.e. training one model per category such as different models for glasses (2 joints) and refrigerators (1 joint) which is a major limitation in previous works.

Weaknesses

1. The paper mentions CARTO, which is super relevant and recent CVPR'23 work but doesn't directly compare to it in terms of both detection and reconstruction. Is there a reason for it? Rather the paper takes individual joint parameter baseline i.e. OPD and shape reconstruction baseline i.e. A-SDF and compares them separately to those. While these are relevant, CARTO is the most relevant to this work in terms of system-level joint detection and shape reconstruction. 2. While the work offers a solution for handling an arbitrary number of parts in the image, the qualitative results show simple examples i.e. same types of joints and 2 joints mostly. Did the authors test their approach on more complicated geometries i.e. varying number of joints or articulated objects with more joints >5+. 3. I didn't see a discussion to run the speed of the model. The most relevant works which the author discusses i.e. CARTO (which builds upon CenterSnap and ShaPO) are all fast approaches. Is 3DETR equally fast and did the author consider adding that backbone i.e. CenterSnap backbone which offers a faster solution + equally good model in terms of accuracy? This is crucial for real-time applications like robotics or grasping etc.

Questions

Please see the questions raised above in the weakness section. To summarize, a comparison to strong and relevant system-level baselines, more qualitative results, and discussion/comparison to speed vs accuracy tradeoff would be useful to answer.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors can only handle cases where instance boundaries are defined so it would fail in scenarios where a door would be attached to a room let's say.

Summary

The paper proposes a method for man-made articulated objects reconstruction from a single RGBD image across different object categories. The method is based on a detect-then-group pipeline, using kinematics-aware fusion for addressing false negatives.

Strengths

The proposed method addresses the challenging problem of man-made articulated object reconstruction from a single RGBD image, estimating at the same time each part's pose. The overall architecture has a somewhat complicated structure with numerous encoder and decoder modules employed. Nevertheless, the design choices are sufficiently motivated in the text and the loss function is quite straightforward given the overall architecture. The experimental evaluation considers both a synthetic (SAPIEN) as well as a real-world (BMVC) dataset. The proposed method achieves improved performance with respect the state-of-the-art on SAPIEN. The ablative study shows the relative contribution of important components. Regarding reproducibility, sufficient details are provided making it easier to reproduce the results (considering also the supplemental material).

Weaknesses

The method seems to be closely related to [20]. I think a more detailed discussion about the relation between the two methods should be included in the Related work section. Related to this, it is not clear why the experimental evaluation does not consider comparison also with [20]. This is important, considering the few baselines available for this task. Alternatively, methods considering multi-view reconstruction on SAPIEN could also be considered to give additional insight on the performance of the proposed method. On a side note, the considered metrics could also include the joint state error. ### Minor comments - L.45: define acronym NMS - L.127: not clear - Fig. 2 is not referenced in text - Although prior work regarding human subjects is briefly visited in the related work, similar work for animals is briefly mentioned but not specified. Works like [R1], [R2] and [R3] could be included to make this aspect of related work stronger. [R1] Ntouskos, V., Sanzari, M., et al. (2015). Component-wise modeling of articulated objects. ICCV [R2] Zuffi, S., Kanazawa, A., et al. (2017). 3D menagerie: Modeling the 3D shape and pose of animals. CVPR [R3] Jiang, L., Lee, C., Teotia, D., & Ostadabbas, S. (2022). Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities. Computer Vision and Image Understanding, 103483.

Questions

- L.126: why P_O both conditions O and is given also as an argument?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are discussed in the text

Summary

The paper proposes an end-to-end trainable method for reconstructing multiple articulated objects from a single RGB-D image, consisting of detecting parts, reconstructing part-level shapes, and estimating poses, bounding boxes as well as kinematic parameters. The parts are grouped into instances later. The authors propose anisotropic scale normalization for shape reconstruction to accommodate various part sizes and scales. Besides, the authors also propose test-time kinematic-aware part fusion (similar to non-maximum suppression) to reduce false positives when multiple detected results are generated and needed to be merged. Evaluation on both synthetic and real data demonstrates the effectiveness of the proposed method.

Strengths

- The paper is clearly written and easy to follow. - The authors propose a clean pipeline for reconstructing articulated objects that can be trained end-to-end and inferred straightforwardly without composing multiple networks.

Weaknesses

Minor typos: - L38: "a end-to-end" -> "an end-to-end" - L219-L220: "and use cosine scheduler to learning rate of 1e-6" sounds a little strange to me.

Questions

1. How do the authors choose the revolute origin? It seems that any point on the revolute axis can be chosen as the origin. I think this question also applies to prismatic joints. 2. Is there any explanation why anisotropic scaling is better than isotropic one? 3. L235-L236: Do you actually mean 20,000 validation images and 168,726 training images? Otherwise, it is strange why more images are used for validation instead of training.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors has addressed the limitations.

Post rebuttal comments

The rebuttal has resolved my questions. Adding limitation examples, and quantitative comparisons to existing state-of-the-art fast reconstruction techniques like CARTO/ShAPO (and mentioning this as a potential limitation as well) would strengthen the arguments in the paper and I look forward to seeing that in the final version. I am happy with the novelty of the paper i.e. part based reconstruction to handle multiple joint types and retain my rating.

Post rebuttal comments

Thanks to the authors for their comprehensive rebuttal. I believe the authors addressed my concerns, and would like to raise my score to weak accept.