3D Indoor Instance Segmentation in an Open-World

Summary

This work tackles the task of incremental object-discovery for 3D semantic instance segmentation. Unlike numerous concurrent works, it is not unsupervised but enables users (or oracle) to label objects that were identified as unknown in each iteration. The method is evaluated on ScanNet200, the paper proposes three new splits/tasks and measures three different metrics as defined in prior work.

Strengths

The proposed approach has significant practical significance as it enables training on a labelled corpora of data and provides the option to users to label the identified unknown objects and then refine the trained model once training labels are provided by a user.

Weaknesses

The manuscript is unclear on numerous occasions to the point that it is hard to understand the method (see questions below, these should be addressed in the rebuttal and incorporated in an updated version). In general, the paper might be clear for those that worked on this project (I assume for a significant amount of time) but the writing is challenging to understand by someone that reads about the project for the first time (unfortunately this makes it really difficult to write a meaningful review).

Questions

Things that are not clear and should be improved: - Looking at the videos and experiments: There are two methods 3D-OWIS-PC-CT and 3D-OWIS - are both yours? Or is one the baseline that is not yours, and the other yours? Maybe you can clearly mark yours with “Ours”. - l.106-110: “the learned that updates itself” -- what does it mean? Is it correct that after one iteration of predictions by the model, the users labels the unknown classes, and then (in the next iteration) the model is trained again based in the existing and additionally provided user annotations? If this is the case, it is confusing to say “continuously improving itself (l.109)” since that somehow suggests that the model can improve without human intervention whereas here it seems to rely on human labels? I do not question the usefulness of user input after each iteration (i think it is very desirable), I simply do not understand what is happening in the end - the description should be improved. - Figure 3 / Sec 3.2 Split B - I do not understand the split - what does it mean “exploring indoor areas” / “accessing an indoor space”? Is there a robot walking around in a simulation (I guess not)? - l.132 - what is “the auto labeler”? The term is used here for the first time without explanation. The text then continues in l.136-143 explaining the drawbacks and proposed solutions. Since I don’t know what an auto-labeler is, these two paragraphs did not provide any information to me so I ignored them for writing this review. L. 144 “the auto labeller depicted in Fig.2” - it is not depicted in Fig2 - there is only a box with the name “auto labeled” - this does not provide any useful information to understand what is happening. - Eq. 2, what is a class prototype? Do we have a class for each semantic class i.e. max of 200 in ScanNet200, or does it refer to individual object instances? The text suggests that it refers to semantic classes. If that is the case, how do you separate different instances that have the same semantic class? - l.175 - what is 3D-OWIS? Is it the same as [20]? After reading l.172-177, they seem to be the same, but I could not find the name 3D-OWIS in [20]. Is 3D-OWIS maybe the name of the method proposed in this paper? The caption of Fig.6 it seems to become clear that 3D-OWIS is the name of the method proposed in this paper. Similarly, in line 244 what is -PC-CT? What are the contributions? What does PC CT stand for? The last line of the caption in Tab.3 indicates that PC is probability corrections dn CT is confidence threshold? [I have to play detective only to understand what is the name of your method, why do you do this to me? Imagine R2 - they will directly reject the paper :( ] - Why are the Tasks 1-2-3 introduced? To my understanding, it seems to be a redefinition of the existing head-common-tail classes. Why confuse the reader and introduce an alias for sth that is already clearly defined?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

No / not applicable

Summary

This paper proposes a pipeline to do 3D open-world instance segmentation. The authors provide a problem definition and introduced three different scenarios. Moreover, to overcome the possible problems that may lead to lower performance on known classes, the authors proposes different modules like probability correction and exemplar replay. The comprehensive experiments show their proposed methods' effectiveness.

Strengths

As claimed by the author, this paper is the first to investigate the 3D open-world instance segmentation task, and develop some well-motivated modules to improve the performance upon the simple baseline.

Weaknesses

I'm not very familiar with the open-world related task and common practices both in 2D and 3D. Currently I think the writing of this paper is confusing and seems to be finished in a hurry. For example, the "3D-OWIS-PC-CT", I assume the "-PC-CT" is **without PC and CT**? And I couldn't find explanations about the "PC" and "CT". In addition, I think authors can also provide the results of their pipeline equipped with other 3D instance segmentation as a comparison.

Questions

As mentioned before I'm not an expert in this field, I'll ask more questions in later reviewing process. Currently I have one concern, since the network won't be re-trained in test and the network structure is fixed, does it mean the total class number (including the known classes and the unknown classes) will have an upper bound? And will the performance of this feature-cluster-based new class discovery become lower as the new class number increases?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

No. The authors didn't address their limitations, and also the possible negative societal impact. I would suggest authors discuss some foundamental drawbacks of current open-world discovery settings.

Summary

This paper introduces an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. Pseudo label quality are further improved subsequently.

Strengths

1. Open world 3d instance segmentation is an important problem and the author first addresses this problem. 2. The method designing is reasonable and complete. 3. The experiment is also sufficient.

Weaknesses

1. Some designs are hard to understand. For example, for probability correction, why unknown object has to be far from the nearest known class? since we have no prior about category distributions. In experiments, you can also see that PC does not bring improvement to all metrics. This should be illustrated more clearly. 2. Although such open world task has not been studied in 3d tasks, it has been widely studied in 2d tasks. Since the core issue of this problem is actually the same, the author should implement some 2d open world methods in 3d instance segmentation and compare with them.

Questions

Please refer to the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author does not discuss the limitation.

Summary

The paper presents a new application of open-world object detection to the setting of 3D instance segmentation. In this setup, a 3D point-cloud segmentation model is required to label each point with instance information, whether this instance is part of training or not. The paper uses ScanNet and proposes a few ways to partition the dataset to use for evaluation in open-world. Finally, the paper adopts the framework in [16] as a baseline for this problem and proposes two tricks on top this framework: PC (probability correction) and CT (confidence thresholding).

Strengths

The task of open-world segmentation in 3D is important yet overlooked by previous work. As outlined by the paper, prior works mainly focus on 2D setups, either image or video. The use of ScanNet for evaluation seems appropriate and the splitting mechanism also seems reasonable, especially the Region-Based split has a very nice motivation from real-world application.

Weaknesses

Post rebuttal comments: 1. The author provided more insights towards the difference between 2D and 3D. I was hoping for more attributes unique to 3D. But having some insights as least improve the quality of the work. 2. My second concern is addressed. Original review: Although there are many things to like about the proposal of the task, this work may suffer from a few important weaknesses: - Missing insights specifically to 3D. The major contribution of the paper is to adopt the recent new task of open-world localization to 3D setting. Unfortunately, the paper does not focus enough on aspects unique to 3D. Most components are the same as the 2D task and insights provided by the paper do not contain enough 3D-specific information. To justify the contribution, it is important for the paper to provides both intuitions and empirical proves on why it is worth studying 3D and how it is different than simple adoption of a 2D framework. In fact, the 3D-OWIS seems to be almost identical to [16] besides swapping the backbone predictor with a 3D backbone. The claimed contributions PC and CT are also not specific to 3D or reveals anything special about 3D. - Lack of comprehensive evaluation/ ablations. If author would prefer to claim more of their contributions on the dataset part, it is important to benchmark the task comprehensively. A suite of recent baselines in 2D should be evaluated. For example, recent works OLN [A] and GGN [B] both provide good class-agnostic proposals, and their 3D variants should be evaluated. In particular, GGN also uses a self-teaching schema. Minor: - The presentation of baseline 2D-OWIS-PC-CT seems really vague when first-time introduced. Better elaborate on what this is when it is discussed the first time. - If author prefers to claim contributions mainly to the task/ benchmark, NeurIPS D&B is more appropriate. [A] Kim et al., Learning Open-World Object Proposals without Learning to Classify, ICRA 2022 [B] Wang et al., Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity, CVPR 2022

Questions

NA

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitation is not sufficiently discussed. For example, when repurposing ScanNet200, what are some potential limitation? This dataset isn't designed specifically for this proposed task in open-world.

Summary

This paper addresses the challenge of 3D instance segmentation in open-world scenarios. It starts with a formulation for this problem, including the definition and setup of known and unknown objects and different splits of categories for simulating different open-world cases. Accordingly, this work proposes a framework incorporating an unknown object identifier to detect objects not present in the training set and devises several strategies to enhance the separation of classes within the query embedding space, such as contrastive clustering and reachability-based probability correction. Experiments on the new benchmark validate the effectiveness of the proposed framework.

Strengths

- This paper studies a new problem setting, open-world 3D instance segmentation, with a new benchmark and framework. - The basic idea is easy to follow and the presentation is overall clear. - The illustration figures are also clear such as Fig. 2-4. - The new benchmark considers different open-world cases when splitting different categories. - The corresponding modifications tailored to this new problem are reasonable, from contrastive clustering for queries to reachability-based probability correction and alleviating catastrophic forgetting for incremental learning. - Experiments show the effectiveness of the proposed framework. Different evaluation metrics also show different perspectives for comparing different methods.

Weaknesses

- There is no preliminary section for the baseline framework, Mask3D, making the introduction of corresponding modifications tailored to this new setting a little confusing. In my opinion, it would be much better to introduce the baseline framework after setting up the benchmark after Sec. 3.2 and even open a new section for the "approach" part. - Some of the modifications are adapted from experiences in the 2D domain, and they are not revised to fit the 3D problem or tackle some specific challenges in 3D. Although some of them such as reachability-based probability correction have their own insight, it is still unclear whether the underlying reason for these challenges is related to the different data modality. It would be better to have deeper thinking from this aspect. - The evaluation benchmark involves comprehensive metrics for the new setup, but it lacks an intuitive connection or comparison between the new problem and other conventional ones, such as 2D open-world instance segmentation/detection and 3D close-set instance segmentation. Given this is a relatively new problem, it would be better to compare it with other well-explored problems to make it more friendly and acceptable for researchers from this community.

Questions

None.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The author does not discuss the limitations and potential social impacts in the paper.

Thanks for your response

After reading the response from authors as well as reviews from other reviewers, I think my initial concerns were shared among other reviewers. Given the response, I think my second concern regarding lack of studies adopting 2D detectors is resolved. It seems non-trivial to extend them to 3D setting, and an naive extension with GGN fails due to the reasons explained by the authors. The author gave some analysis regarding 3D features being more scattered and difficult, and I think this insight is useful. However, I think what (other reviewers and) I are looking for, is something unique to 3D, such as geometry. For example, the failed adoption of OLN due to sparsity of point-cloud is quite interesting. Some other angles relating to 3D geometry might provide extra insights to strengthen the paper. For example, is 3D shape harder to generalize in open-world than 2D? Or maybe shapes are less prone to overfitting in closed-world scenario. That being said, I think the author did a good job responding to the concern regarding adopting other 2D detectors. I am inclined to raise my rating to borderline accept or weak accept. Would love to hear from other reviewers' post-rebuttal comments.

Thanks for the rebuttal!

I thank the authors for the extensive rebuttal which provided insights to my remaining questions. An updated version can be improved by including those clarifications as well as the additional experiments.

Response to Rebuttal

I acknowledge that I have read the authors' rebuttal and the other reviews. Thank you for addressing my concerns. I think most of my concerns are addressed to some extent and will keep my score in the final decision. I strongly recommend the author carefully consider the reviewers' comments and revise the paper as promised in the rebuttal.

Thanks for the rebuttal

Thank you for your rebuttal. Most of my concerns are addressed, and I will keep my score.