LESS: Label-Efficient and Single-Stage Referring 3D Segmentation

Response to Reviewer NB5W

Thank you for your questions and suggestions. A recent work, OpenScene[1] has established a straightforward and efficient baseline network in the field of open vocabulary 3D segmentation using a simple cross modal distillation method based on primary cosine smilarity driven, but also with a pioneering explroation. The objective of our work is to present a straightforward and effective single-stage baseline for the task of 3D referring segmentation. In addition to a single stage architecture design, our work makes a further contribution in the form of a single-stage coarse-to-fine cross-modal alignment network and a coarse-to-fine background point filtering loss function. These have been developed with the aim of achieving state-of-the-art performance under efficient binary mask supervision, which have been demonstrated with extensive ablations mentioned before. Thank you again for your suggestions. We sincerely hope that you will be able to evaluate the contribution of our work based on its overall design and the differences that it presents in comparison to previous works [2][3], and also the hitting cornerstone of single-stage pipeline of this task. [1]. Openscene: 3d scene understanding with open vocabularies, CVPR 2023. [2]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021 [3]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks, AAAI 2024

Response to Reviewer NB5W

Thank you for your response! We are grateful and pleased that our response can address most of your concerns. Regarding the novelty of our work, we would like to answer from the following points and discuss the differences from previous works. - The novelty of the Query Mask Predictor and Query-Sentence Alignment. As we explained in the paper, QMP can provide candidates of the target in the scene. However, different candidates have different importance to the target and most of them contains unrelated background points. To address this issue, we propose the QSA module to weight the mask and query by calculating the relationship between the candidate and the query through the cosine similarity. Moreover, **different from CLIP[1]** use cosine, which selects the mask with the highest score as the final result, our final mask prediction is facilitated by a weighted sum of the similarity score and mask prediction. Using a weighted sum enables a soft decision-making process rather than a hard and binary choice. This can help in cases where multiple masks have similar scores, allowing the final prediction to be an aggregation of these maks, which can be more accurate than any single one. As shown in Tab.x, we also conduct related experiments about this, which also confirm the advantages of our method. **In comparison to Mask3D [2]**, our QMP and SQA generate possible mask proposals guided by the language caption, whereas Mask3D's decoder enumerates all possible proposals in a scene without any guidance. | | mIoU | Acc@25 | Acc@50 | | :------: | :-------: | :-------: | :-------: | | Top-1 | 33.18 | 52.93 | 28.65 | | Baseline | **33.74** | **53.23** | **29.88** | - The novelty of the Point-Word Cross-Modal Alignment. The PWCA is based on the cross-attention structure, which is a prevalent approach in cross-modal fusion networks. **In comparison to the original cross attention model[3]**, we have removed the position encoding in order to address the disorder of 3D point clouds. Furthermore, we have incorporated an MLP Tanh gate on top of the original model in order to regulate the information flow between language and visual features. - The novelty of module design. In contrast to previous research, which solely aligned word features with proposal features, we introduce a novel one-stage coarse-to-fine point cloud language cross-modal alignment paradigm. PWCA, QMP and QSA are design to align word-point and sentence-mask, respectively. Furthermore, another significant innovation is the introduction of a coarse-to-fine background point filtering strategy, comprising Area Regularization Loss and Point-to-Point Contrastive Loss, which enables the effective processing of high-complexity, large-scale 3D scenes. - Finally, related experiments shown that compared to previous works, our LESS is simple yet effective not only in SoTA performance but also in efficient time consumption with only binary mask supervision. As mentioned in conclusion of our paper, we aims to construct a straightforward and universal single-stage baseline to establish a solid foundation for this task, rather than focusing solely on the performance enhancement offered by a single novel model architecture, which sheds new lights on future exploration on how to better align the visual feature with language feature (cross-modal) or propose different training strategies. We would like to express our gratitude for your valuable suggestions and for highlighting the existing issues. Hope that our response could provide the necessary clarification to address your concerns. If you have any additional questions or suggestions, we are very glad to have a further discussion. [1]. Learning Transferable Visual Models From Natural Language Supervision, ICML 2021. [2]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023. [3]. Attention is All You Need, NeurIPS 2017.

Response to Reviewer 6zjS

Thanks for your thoughtful reflections on this question and pointing out the confusion about the calculation and comparation of label efficiency. We will discuss these questions more deeply and below are our analysis and explanation: ## Q1: Rethinking whether not using the AP metric in instance segmentation - **Why the previous works[1,2] are called "instance segmentation" :** As we mentioned in the paper, the previous works adopt two-stage workaround. The first stage utilizes a 3D instance semantic segmentation network to get instance proposals. And the second stage leverages a network to match the query with corresponding instance proposals. Because the first stage is of great importance to this task, previous works simply named this task with "instance segmentation", while we followed the previous works to do that and will extand more discussions into our revised version to rethink how to strictly name this task. - **Why not use AP metric in the R3DIS task:** Thanks for this detailed reading and question. The target of referring 3D instance segmentation about a corresponding query is only one. Therefore the IoU metric is sufficient to measure this task. In addition, the Acc@x metrics also is used to measure the accuracy of the segmentation mask. We will consider to add more comparative definition between Acc@x and AP into our revised implementation details. - **Rethinking about our LESS:** We will take necessary in-depth discusions to explain and compare referring 3D instance segmentation with conventional instance segmentation task. Since our method is one-stage and lacks a first instance proposal stage, the title with "instance segmentation" may be not strictly align with the common instance segmentation task. However, the core contribution of our paper is to develop a simple yet effective single-stage for this community, shifting from complicated two-stage pipeline into single-stage one, which paves more spaces for future works. We will add more in-depth discussions to the final version of the paper and consider whether to revise the confusing title. ## Q2: For the labelling time on the whole dataset - **The number of scenes needed to annotate:** Previous works [1,2] contain a 3D instance segmentation and a matching network. Based on their open source codes, we find that training a typical instance segmentation network requires annotations for **1513** scenes (train+val) rather than **800** scenes (which is the number of ScanRefer). Therefore, in light of the aforementioned assumption that the time required for the annotation of a single scene is 25 minutes, the overall time required for the training process should be 25 * 1,513 = 37,825 minutes, rather than the 20,000 minutes as estimated before. - **The number of objects needed to annotate:** In order to make a fairer and more reasonable comparison of label-efficiency between our LESS and previous works, we compare the number of objects required labelling. Firstly, we follow the open source codes of the previous works to process the dataset. From the processed dataset we find that the number of annotated objects under 1513 scenes is about **45,711**, even though excluding walls and floors. As for our LESS, we only need to annotate **11,046** objects, which is 4 times fewer compared to previous works. Here we need to note that the objects referred by ScanRefer is the subset of the objects in ScanNetv2. - **For the comparative method of time in the paper:** As we mentioned above, the objects referred by ScanRefer are a subset of all objects among all scenes. In the context of the time comparison presented in the paper, the comparison of labeling effort is based on a single sample. This is because that, in consideration of reality scenarios, it is often the case that the model is adapted to incorporate new data in order to align with evolving referential requirements. Therefore, in our method, there only necessitate just a single or a few objects needed to be labelled, rather than the entire scene. When scaling up the dataset scale for this task, our advantages will become much more significant. As a result, we compared the label effort of a single sample in the paper. Thank you for pointing out this issue, and we will make further discussion on this in our paper to reduce any potential ambiguity. In summary, our method takes less time than the previous works in different comparison methods, which indicates our method is more label-efficient. Also We would like to extend our gratitude once more for your insightful suggestion. It is our sincere hope that the responses above could address your concerns. If you have any additional questions or suggestions, please let us know. [1]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021 [2]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks, AAAI 2024

Response to Reviewer Monv

We are grateful for your comprehensive and encouraging review and acknowledgement. We are pleased that you appreciate the technical contributions of our work, which could motivate us to explore more interesting works in the future. Thank you again for your support!