OpenMask3D: Open-Vocabulary 3D Instance Segmentation

Summary

This paper focuses on the task of open-vocabulary 3D instance segmentation, which involves predicting 3D object instance masks and their corresponding categories. The authors highlight the limitations of traditional closed-vocabulary approaches that operate within a predefined set of object categories, which restricts their ability to handle novel objects and free-form queries. To address these limitations, the authors propose OpenMask3D, a zero-shot approach for open-vocabulary 3D instance segmentation. OpenMask3D utilizes predicted class-agnostic 3D instance masks and performs mask-feature aggregation using CLIP-based image embeddings. This allows the model to reason beyond pre-defined concepts and handle open-vocabulary queries. This paper demonstrates its superiority over closed-vocabulary counterparts. The proposed method has the potential to enhance various applications such as robotics, augmented reality, scene understanding, and 3D visual search.

Strengths

The authors propose OpenMask3D as the first zero-shot approach for open-world 3D instance segmentation, offering a unique solution that leverages predicted class-agnostic 3D instance masks and CLIP-based image embeddings. This combination of ideas and techniques demonstrates originality in problem formulation, model architecture, and feature aggregation for open-vocabulary 3D instance segmentation. The methodology is presented in a clear and structured manner, allowing readers to understand the proposed approach easily. The experimental evaluation is conducted on the common benchmark datasets, and the authors perform ablation studies to gain insights into the design choices of the model. The results demonstrate that OpenMask3D outperforms other open-vocabulary counterparts, particularly in scenarios with a long-tail distribution of objects, which advance the state-of-the-art 3D instance segmentation in the complex real world.

Weaknesses

While the paper demonstrates several strengths, there are also some areas that could be improved to further enhance its contributions and impact. Evaluation on a Single Dataset: The paper conducts experiments and ablation studies on the ScanNet200 dataset, which may limit the generalizability of the findings. It would be beneficial to evaluate the proposed OpenMask3D method on multiple datasets, including those with varying object distributions, to assess its robustness and performance across different scenarios. A more diverse evaluation would strengthen the claims made in the paper. Long Pipeline: another weakness of this work is the long pipeline of the proposed OpenMask3D model. The pipeline consists of multiple stages, including a class-agnostic mask proposal head, mask-feature aggregation module, and multi-view fusion of CLIP-based image embeddings. The length of the pipeline may introduce computational complexity and potentially impact the efficiency and real-time applicability of the model. Lack of Computation Cost Comparison: While the paper discusses the architecture and methodology of OpenMask3D in detail, it does not provide a thorough comparison of the computational resources required and the speed achieved between different methods. Problematic Structure of section 4.1: sec 4.1 is named as "Closed-vocabulary 3D instance segmentation evaluation", while both closed-set instance segmentation results and open-vocabulary instance segmentation results are presented in this section.

Questions

The paper primarily evaluates the proposed OpenMask3D method on the ScanNet200 dataset. Evaluation on more datasets with diverse scenes should be performed. It would be beneficial to include a comprehensive comparison of the computational resources and inference speed between OpenMask3D and existing closed-vocabulary and open-vocabulary 3D instance segmentation methods. The structure of Experiment section should be better organized.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper briefly mentions limitations of OpenMask3D. It would be valuable to provide a more thorough analysis of the limitations, and discuss potential method / direction that could be explored to solve it. Potential negative societal impact is not applicable for this work.

Summary

The paper proposes to perform open-vocabulary instance segmentation by utilizing class-agnostic 3D instance segmentation masks from a 3D instance segmentation model trained on scannet200 and generating class labels for it using CLIP. The paper proposed to first obtain class-agnostic instance masks from a supervised Mask3D model (without using class annotations). Then it finds the images where the objects are best visible. It then projects the 3D segmentation masks in those views and refine them using SAM. Finally they average features for each object from multiple views and at multiple scale to arrive at one feature vector per 3D instance mask. The class label can then be obtained by doing dot-product with the obtained feature vector and the language embedding of the class. The results show that the proposed method is superior than prior point-based methods when also supplied with class-agnostic instance segmentation mask in closed-vocabulary setting. The paper also shows some qualitative results with free-flowing natural language.

Strengths

- The paper is well written, I especially appreciate the helpful supplementary video and text content which made the nitty gritty details of the pipeline very clear. - Open-Vocabulary 3D instance segmentation is a very useful task which hasn't been tackled before -- this paper brings attention to it (that said, I have some concerns here as mentioned in weaknesses) - The proposed method obtains better results than prior point-based methods like OpenScene

Weaknesses

- Open-Vocabulary: The proposed method relies on 3D instance segmentation predictions which in turn relies on 3D segmentation annotations. This, however, is not available for wide variety of objects, hence I am unsure if we can conclude that this model is indeed “open-vocabulary”. For example: If instead of scannet200, the proposed model uses class-agnostic masks from a model trained on 20 scannet classes, would it be able to achieve decent results on Scannet200? Would this model trained on scannet work on a different dataset like MatterPort3D? Additionally, since 3D datasets are significantly smaller than their 2D counterparts, doesn’t relying on 3D instance segmentation mask a serious bottleneck which wouldn’t scale? The point-based models are open-vocabulary in the sense that they are not bottlenecked by any 3D-specific annotations — at the same time I do agree with the point of this paper that they can only do semantic and not instance segmentation. However, using instance masks via 3D annotations might not as well lead to “open-vocabulary instance segmentation” proposed in this paper. Ideally, it should primarily have results on held out 3D categories that the model or any of its components have never seen during training. - In continuation of the above, the results for 3D instance segmentation in open-vocabulary setting is only qualitative and not quantitative. I understand though that prior methods too show only qualitative open-vocabulary results, and so this is said as a minor point and not a major complaint. However, the lack of quantitative results on categories outside the training data is concerning — especially since this proposed model particularly used labels from scannet which may not generalize beyond the 200 categories they were trained on, and on out-of-domain 3D scenes. - Unfounded Claims: - L304: The paper highlights that when given access to oracle masks at “test time” to their model, it outperforms the supervised Mask3D model on tail AP by 9.1%. As the paper concludes in L305-308, this result indicates that if somehow we are able to obtain high quality class-agnostic masks, we do not need supervision for class labelling as their method can outperform supervised Mask3D. In my opinion, this is a misleading claim because while their baseline “Mask3D” has access to ground truth masks and classes during training, it does not have access to oracle masks during test time. This makes the comparison unfair. AP is very sensitive to quality of the segmentation mask and if supplied oracle masks to Mask3D it may do much better. A very crude way to supply that would be — computing Hungarian matching between the predicted masks and ground truth masks (without class labels), and for each predicted mask replace it with the matched ground truth mask while keeping the class label same. In general though, this comparison of supervised mask3d and the proposed method needs much more care to make balanced conclusions. - (Minor) L290-291: Claims that “the ablation study show that effect of 2D mask segmentation is less significant than the effect of multi-scale cropping”. Based on this, one might expect the row 2 of component analysis section to be (significantly) better than row 3. However, on some metrics row 2 wins while on others row 3 wins. And the difference is not that much.

Questions

- As mentioned in the limitations, here are some suggestions which might help alleviate the concerns over instance-segmentation annotation bottleneck: - Does class-agnostic segmentation generalize beyond training categories? Performance on OOD dataset like Matterport3D and maybe instance segmentation model trained on scannet 18 classes and tested on scannet 200 class could help (compared against point-based methods). Another suggestion could be to hold out rare categories from Scannet200 for training class-agnostic instance segmentation model and evaluate on the held out categories at test time. Ofcourse, these are just suggestions and any other experiment that could help us get to the bottom of this will be highly appreciated. - Answers to "unfounded claims" as described in limitations would be super helpful too. - Could you give some insights on why is the performance of OpenScene 2D Fusion model so much better than 2D-3D ensemble models while the Openscene paper consistently showed better results with the ensemble version of their model? That was a bit strange to me.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper do not discuss limitations. I think the biggest one is their reliance on class-agnostic instance segmentation mask annotations which would be good to discuss in the ppaer.

Summary

This paper presents a method for 3D open-vocabulary instance segmentation. It proposes to use a class-agnostic Mask3D to get some instance mask proposals, project the instance points to 2D views to get some 2D segments, and extract some instance features based on the 2D segments by CLIP image encoder. Then, text descriptions can be used as queries to find the best instance proposals in an open-vocabulary way by comparing the similarities with the instance mask features.

Strengths

1. The paper is well organized with good figures and easy to understand. 2. The paper is the first method for open-vocabulary instance segmentation in 3D scenes. 3. The experiments and analysis in the paper show the great zero-shot ability of the method.

Weaknesses

1. The method relies on RGB-D images, and thus cannot generalize to 3D scenes (e.g., 3D scenes collected by LiDAR) without 2D images. 2. The experiments are conducted only on the ScanNet dataset. Experiments on more datasets like the Matterport3D used in OpenScene are expected to prove the generalizability of the method. 3. The inference speed of the whole framework seems slow as the method uses some heavy models (e.g., SAM and CLIP) for multiple views. SAM model is even used multiple iterations for each mask. A comparison of the inference time between this method and OpenScene is expected.

Questions

Please refer to the weaknesses. More questions are listed below: 1. More details of training a class-agnostic Mask3D are expected, for example, the scores used in the Hungarian matching process and the losses used in training. 2. The paper will be better if there is a discussion of the domain gap between the cropped image and the images used to train CLIP. Also, considering the domain gap, a detailed ablation of the hyperparameters in multi-scale cropping is expected. 3. The equations in L212 seem not accurate.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Some of the limitations are discussed in the last paragraph of the paper.

Summary

This paper addresses the problem of text-based 3D instance segmentation. To tackle this problem, the authors propose OpenMask3D, a zero-shot approach for 3D instance segmentation that utilizes class-agnostic 3D instance masks and multi-view fusion of CLIP-based image embeddings to aggregate per-mask features. The model's performance is evaluated through experiments and ablation studies on the ScanNet200 dataset, where it outperforms OpenScene in some metrics.

Strengths

**Clarity and quality**: The paper is well written and explains all the components with clarity and in detail. The figures are excellent. The results are very neatly presented and overall the paper is engaging to read. **Empirical Evaluation**: The paper includes extensive empirical evaluations on the ScanNet200 dataset. The paper not only demonstrates the performance of OpenMask3D but also presents ablation studies on the number of frames used, the use of 2d masks and multi-scale crops to understand the contribution of these components. **Relevance**: The work addresses a highly relevant challenge in computer vision and autonomous systems. As these technologies become increasingly prevalent, the ability to understand and interact with a diverse range of objects becomes critically important, underscoring the relevance of this research.

Weaknesses

**Novelty of the task**: The paper makes strong claims about being the first to introduce "open-vocabulary 3D instance segmentation". However, given OpenScene and others I don't think it can be claimed that the problem of text-based 3D instance segmentation is novel. OpenScene might not have used the word "instance" but it does show results for "Open-vocabulary 3D object search" which is similar in meaning. **Significance of the Contribution**: It is true that most existing segmentation approaches have relied on a closed-set of objects in 3D annotated datasets and thus a system that can perform open-vocabulary segmentation would be quite a significant contribution to the field of 3D scene understanding. However, the technical contribution of the proposed approach is somewhat limited considering its similarity to OpenScene. The main differences seem to be the frame selection and feature aggregation strategy, but if this is the case I would expect these components to be more prominently handled in the paper (rather than claiming a whole novel task).

Questions

- Please clearly outline the differences compared to OpenScene - The examples given in the paper are mainly for common objects "armchair", "seat", "pool", "sofa". Can you give an example where the method identifies objects that are truly from an "open" vocabulary (i.e. not common)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper proposes to solve open-vocabulary 3D instance segmentation. It uses a class-agnostic 3D instance segmentation model to obtain instance masks, then gather multi-scale image features from multiple frames by CLIP and SAM to do the open-vocabulary classification task.

Strengths

1. The paper proposes to use instance-level features for 3D open-vocabulary instance segmentation, which is not attempted by previous works 2. The proposed module exhibits reasonable improvements, as shown in Tables 2&3.

Weaknesses

1. The design of the framework may be complicated for real-world usages. It uses SAM to do segmentation for multiple frames, and then use multi-scale images for CLIP to inference. Each component like SAM and CLIP is a large foundation model and takes a while to inference, not mention that they are used multiple times. 2. The idea is not so novel. Although such idea of using instance mask features is not attempted in 3D instance segmentation, it has been widely adopted in 2D open-vocabulary segmentation tasks, like OpenSeg[1], ODISE[2], ZegFormer[3]. 3. The experiments are not thorough. For example, the details of using SAM and RANSAC are not studied. [1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV2022 [2] ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models, CVPR2023. [3] Decoupling Zero-Shot Semantic Segmentation.

Questions

1. What is the latency and inference cost of the model during inference? 2. How $k_{rounds}$ is chosen? Is there any ablation study about that? Furthermore, does it mean that SAM needs to inference $k_{rounds}\times number of frames$, and CLIP need inference $k_{rounds}\times number of frames \times number of scales$ 3. How the confidence score $s_r$ is obtained?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

See weakness