RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

Summary

This paper presents RG-SAN, a new method for 3D Referring Expression Segmentation. It combines spatial reasoning with textual cues to segment 3D objects accurately. RG-SAN uses a Text-driven Localization Module and a Rule-guided Weak Supervision strategy. It outperforms existing methods on the ScanRefer benchmark. It handles spatial ambiguities well and sets a new precision standard for 3D scene understanding. Extensive ablation studies and visualizations validate the effectiveness of the proposed modules.

Strengths

1) The motivation of this paper is clear. Figure 1 shows how spatial relationship info in natural language matches with 3D scenes, making the motivation obvious. Figure 4 reinforces this motivation from a quantitative perspective through statistical analysis. 2) I agree that spatial relationship reasoning is crucial for understanding 3D scenes. The complex spatial relationships in natural language give important spatial clues. The proposed Text-driven Localization Module (TLM) performs spatial reasoning explicitly, aligning with how humans understand 3D scenes, which makes sense to me. 3) The Rule-guided Weak Supervision (RWS) module enables localization and segmentation of auxiliary object nouns in natural language without any supervision. This aspect is interesting and shows the model's generalization capability. 4) The appendix includes an analysis of how well large language models (LLMs) can localize target words, comparing this with the RWS module's results. This comparison further validates RWS and offers new insights into using LLMs for the 3D-RES task. 5) The paper reports extensive experiments on common 3D-RES datasets like ScanRefer and ReferIt3D, achieving state-of-the-art performance. Ablation studies also show the effectiveness of the TLM and RWS modules

Weaknesses

1) The paper does a good job comparing 3D-RES methods. But traditional 3D Visual Grounding methods using bounding boxes are more mature. I'd like to see how these older methods perform on this task. This would give a more complete quantitative comparison with the proposed method. 2) The appendix talks about LLMs for target word localization. But it doesn't compare them directly with 3D Visual Grounding methods based on LLMs. Could LLM-based approaches be better? I'd like to see a comparison between specialized lightweight models and general LLMs. 3) The paper should provide details on the superpoint feature extraction mentioned in line 119. I'm curious if superpoint features cover all targets. If not, what's the missing rate? 4) The layout of Tables 2 and 3 is off, and the font in Figure 2 is too small. This makes it hard to read the details. 5) The text processing procedure isn't detailed enough. For example, the interaction process of the DDI module. I'd recommend including this description in the main text.

Questions

1 Will the code for RG-SAN be open-sourced in the future? 2 Could the authors provide a quantitative comparison of RG-SAN with a broader range of 3D visual grounding methods? Others please look at weaknesses.

Rating

8

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations are discussed [line 576].

Summary

This paper presents a novel and high-performing 3D referring segmentation network. Specifically, it approaches the problem from both 3D spatial relationships and natural language spatial descriptions, innovatively using explicit spatial position modeling and multimodal interaction. This allows the query corresponding to textual entities to understand both semantics and spatial locations. Additionally, the use of weak supervision techniques enables the model to achieve strong generalization capabilities even under incomplete annotations. Comprehensive experiments further validate the superior performance of the proposed method.

Strengths

I commend the authors for their insightful paper, particularly the proposed text-guided spatial perception modeling approach. This method aligns with human cognitive habits and has the potential to significantly advance the field of multimodal 3D perception. Several notable advantages are highlighted: 1. The paper deeply explores text-conditioned 3D spatial perception from both 3D spatial relationships and natural language structure perspectives, advancing the community's exploration of multimodal spatial perception modeling. 2. The proposed TLM module effectively addresses the challenge of explicit spatial reasoning in previous end-to-end segmentation paradigms, significantly improving segmentation performance while maintaining high inference speed. 3. The RWS module demonstrates data efficiency, generalizing capabilities to all entities without requiring mask labels for all textual entities. 4. The experiments are comprehensive, evaluating the model's performance on both the ScanRefer and ReferIt3D datasets, thoroughly validating its robust performance. 5. The ablation studies are detailed, thoroughly analyzing the proposed TLM and RWS modules, as well as the backbone selection and hyperparameter settings. 6. The video demo in the open-source link is engaging, and the visualizations in Figure 3 of the paper are intuitive, effectively illustrating the core ideas and powerful performance of the proposed method.

Weaknesses

1. More details can be added regarding the superpoint feature extraction and text feature processing sections. 2. The paper mentions that the Sparse 3D U-Net used as the visual backbone is pre-trained. On which datasets was it pre-trained? Would using different pre-trained backbones result in performance variations? 3. Figure 2 has too many colors, making it somewhat cluttered and potentially confusing for readers. It is recommended to simplify and optimize the color scheme. 4. It is suggested to include some bad cases to enhance the completeness of the work.

Questions

Will the complete code for the paper be open-sourced for additional exploration?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors discuss the limitations in the appendix.

Summary

This paper presents the Rule-Guided Spatial Awareness Network (RG-SAN) for 3D referring expression segmentation (3D-RES), offering a novel approach to understanding spatial relationships in the visual-language perception domain. It aligns 3D and linguistic features not only at the semantic level but also within geometric space. The proposed network incorporates modules for textual feature extraction, text-driven localization, and rule-guided weak supervision. In the experimental setup, the model builds upon the efficient Superpoint Transformer for 3D feature extraction, as developed by Sun et al. (2023). The experimental results are promising, particularly in the overall 0.25 threshold setting on the ScanRefer dataset. Extensive ablation studies demonstrate the effectiveness of the proposed modules, while vivid visualizations showcase the method's impressive generalization capabilities.

Strengths

1. RG-SAN approaches 3D-language semantic alignment and spatial perception from a novel perspective. It not only aligns language features with 3D point cloud features at the semantic level but also explicitly assigns spatial positions to textual entity words within the geometric space. This explicit alignment helps address the spatial ambiguity inherent in directional natural language, allowing RG-SAN to achieve a more precise understanding of spatial relationships described in text. 2. The authors conducted comprehensive experiments on the 3D-RES task, with particularly notable performance improvements on the ScanRefer dataset. It is impressive that the method significantly enhances performance while maintaining rapid inference speed, which is beneficial for real-time applications of this task. 3. The authors performed detailed ablation studies on the proposed TLM and RWS modules, thoroughly examining the settings and hyperparameter choices. Additionally, they conducted ablation studies on the visual backbone and text backbone in the supplementary materials. These comprehensive ablations help readers understand the efficacy and rationale behind the proposed modules. 4. In the supplementary materials, the authors provided a statistical analysis of the importance of spatial information for the 3D-RES task. This quantitative analysis supports the motivation of the paper and gives readers a clearer understanding of the role of spatial information in this task. 5. The authors' visualizations are illustrative. In particular, Figure 3 demonstrates RG-SAN's text-guided spatial understanding and localization capabilities, showcasing excellent generalization. 6. The authors have committed to open-sourcing their method, providing a link that includes the source code and an engaging video demo. This openness will promote development and knowledge sharing within the community.

Weaknesses

1. RG-SAN adopts superpoints as fundamental visual units for feature extraction and segmentation. While previous works have employed similar approaches, I am interested in understanding the segmentation quality of superpoints themselves. For instance, how many superpoints exist solely within individual objects? This is crucial because if a superpoint spans across two objects, it inevitably affects the segmentation results. 2. RG-SAN trains spatial awareness networks using the centroid coordinates of target objects. Here, does "object center" refer to the geometric centroid or the center of mass (where the former denotes the center of the bounding box and the latter denotes the mean coordinate of all points belonging to the object)? Given the inherent sparsity of point clouds, these two centers may exhibit significant differences. 3. The statistical analysis of the importance of spatial information for 3D-RES should be included in the main text. This will help readers understand the motivation of the paper from both qualitative and quantitative perspectives. 4. Although this paper discusses its limitations, it does not provide failure cases or corresponding analyses. Specifically, for the segmentation of plural nouns, it remains unclear whether only one object is recognized or if segmentation fails altogether. It would be beneficial to include either qualitative or quantitative analysis in this regard. Including this part would make the paper more comprehensive and facilitate follow-up research and improvements.

Questions

1. Does "object center" in this paper refer to the geometric centroid (center of the bounding box) or the center of mass (mean coordinate of all points)?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have discussed the limitations, and I think it is somewhat okay for this work. It would be even better if an analysis of bad cases could be included.

Summary

The paper proposes a new framework for 3D referring expression segmentation. The main contributions include analyzing the spatial information among objects and rule-guided target selection. Extensive experiments validate the effectiveness of the proposed method.

Strengths

The authors develop the method to achieve the state of the art performance on ScanRefer benchmark for 3D Referring Expression Segmentation. The authors conduct detailed experiments and comparison to validate the design.

Weaknesses

1. Both the idea of incorporating the spatial information is not new. For spatial relation, section 3.2.2 and section 3.2.3 are very similar to [22]. Equation (7) to equation (10) are almost identical to equation (5) to equation (7) in [22]. Equation (12) is similar to equation (8) in [22]. What's more, there have been a lot of work on spatial information in 3D visual grounding, such as [6]. 2. The writing needs improvement. Some notations are not well explained. For example, K_i in line 120 and c_i in line 123 are not introduced. P^t in equation (3) with two subscripts is inconsistent with P^t in equation (5). Table t in line 166 and q in equation (7) are not introduced.

Questions

1. The text-driven localization module is similar to the architecture in [22]. Have you tried to report their performance in ScanRefer in table 1? 2. In equation (2), how do you initialize W_E and W_S to obtain the initial representations? 3. In terms of equation (5), what is the intuition of adding positional encoding to position features? Does the order of the visual feature affects the final attention output? 4. In equation (6) the positional encoding is added while in equation (9) and (10) it is concatenated. Why are these different? 5. For section 3.3.1, do you have separate evaluation on how your algorithm performs in terms of finding the target? 6. The explanation for table 4 points out the Top1 tends to select different nodes variably (line 281), but RTS is also choosing different nodes?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors addresses the limitations and societal impact.

Sincere Request for Further Discussions

Dear Reviewer 6dEW, Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. Regards, Authors

Sincere Request for Further Discussions

Dear Reviewer p7dx, Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. Regards, Authors

Sincere Request for Further Discussions

Dear Reviewer MnE2, Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. Regards, Authors

Sincere Request for Further Discussions

Dear Reviewer 8gkL, Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. Regards, Authors

Raise the score to 8

Thanks to the authors for an excellent rebuttal—I'm pleased to say that all of my concerns have been thoroughly addressed. The inclusion of both traditional visual grounding and the latest LLM-based approaches really strengthens the paper's conclusions. I'm particularly excited about the LLM-based approach, and I believe expanding on this in future versions could really push the field forward. It’s clear that this aspect has a lot of potential to guide future research. I also took the time to review the other reviewers’ comments, and I stand by my initial impression. The authors' exploration of how spatial relationships in natural language correspond with 3D scenes tackles a crucial and challenging area, especially compared to purely visual 3D segmentation. Spatial and relational reasoning is one of the major hurdles in cross-modal 3D vision today, and it’s great to see the authors making strides in this direction. I’m confident this work will inspire further progress in embodied intelligence. I will fully support this paper and raise its score.