OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

Summary

This paper presents OpenGaussian, a method utilizing 3D Gaussian Splatting for open vocabulary comprehension at the 3D point level. It addresses the limitations of current 3DGS methods that are confined to 2D pixel-level analysis and are inadequate for 3D point-level tasks. The method leverages SAM masks to maintain 3D consistency, implements a two-stage codebook for feature discretization, and proposes an instance-level 3D-2D feature association approach, demonstrating effectiveness in various tasks.

Strengths

1. The motivation behind this paper is well-grounded, and the proposed method is technically sound. 2. The paper is well-written and easy to follow. 3. The method demonstrates good performance across various tasks, including open-vocabulary object selection, click-based 3D object selection, and 3D point cloud understanding.

Weaknesses

1. The ablation studies are not sufficiently thorough. In Table 3, using xyz information results in significant performance improvement. It would be beneficial to provide results for the coarse-level alone, using xyz information, with k=64 and k=320. This would further illustrate the necessity of using the fine-level. 2. A sensitivity analysis for k1 and k2 could be provided to better understand their impact on the performance of the method.

Questions

See weaknesses

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

In this paper, the authors propose three techniques to enhance the point-level 3D gaussian-based open vocabulary understanding: 1. Intra-mask smoothing loss to draw features within the same mask closer, and inter-mask contrastive loss to increase discriminativeness of the mean feature of each instance. 2. Two-level codebook for discretization. The proposed codebook discretizes instance features with the 3D coordinates to ensure identical Gaussian features from the same instance, in a coarse-to-fine manner. 3. Instance-level 2D-3D association technique to link CLIP features with 3D instance without loss backpropagation and depth information.

Strengths

1. Visualizations are clear and strong to show the effectiveness of the proposed OpenGaussian and advantages over previous literature. 2. Quantitative results also demonstrate consistant and remarkable improvements over previous methods.

Weaknesses

1. Paper writing needs to be improved. I have to admit that I'm not expert of this field, and this paper is not easy to understand since it requires abundant prior knowledge about the task and previous methods. 2. No limitation discussion is included in this paper. 3. Ablations on inter/intra-mask smoothing loss, since this is also a contribution of this paper. 4. Efficiency comparison between OpenGaussian and previous methods. The authors do includ training time in the supplemental material. However, since there is human-computer interaction in this task, it is critical to reveal the inference time or throughput of the method. Can it achieve real-time performance? Does it lags behind previous methods in terms of efficiency? 5. The three contributions of this paper seems separate. The authors are encouraged to resummarize them in a more integrated story.

Questions

1. typo: m(m+1) -> m(m-1) in equation (2).

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper doesn't discuss any limitations.

Summary

This paper introduces "OpenGaussian," a novel method for 3D point-level open vocabulary understanding using 3D Gaussian Splatting (3DGS). The authors address the limitations of existing 3DGS-based methods that primarily focus on 2D pixel-level parsing. OpenGaussian aims to enhance 3D point-level understanding by training instance features with 3D consistency and proposing a two-stage codebook for feature discretization. The method also introduces an instance-level 3D-2D feature association to link 3D points to 2D masks and CLIP features. Extensive experiments demonstrate the effectiveness of OpenGaussian in various 3D tasks, and the source code will be released.

Strengths

- **Novelty**: The paper introduces a unique approach to 3D point-level open vocabulary understanding, which is a significant advancement over existing methods that focus on 2D pixel-level parsing.   - **Technical Contributions**: The proposal of a two-stage codebook for feature discretization and the introduction of a 3D-2D feature association method are innovative and well-executed.   - **Experiments**: The extensive experiments, including open vocabulary-based 3D object selection and 3D point cloud understanding, validate the effectiveness of the proposed method.   - **Clarity**: The paper is well-written and clearly explains the methodology, making it easy to follow the proposed approach and its benefits.

Weaknesses

- **Limitations Discussion**: The paper does not discuss the limitations of the proposed method in detail, which could provide a more balanced view of its applicability and potential drawbacks.   - **Comparative Analysis**: While the paper compares OpenGaussian with LangSplat and LEGaussians, additional comparisons with other state-of-the-art methods in OV 3D understanding could strengthen the evaluation, like Open-vocabulary 3D object detection[1, 2]   - **Complexity**: The implementation details, especially the two-stage codebook and feature association, may be complex and could benefit from further simplification or more detailed explanations for reproducibility.   - **Generalization**: The experiments are conducted on specific datasets, and it's unclear how well the method generalizes to other types of 3D scenes or datasets. [1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3 [2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023

Questions

- **Scalability**: How does the method perform on larger and more complex 3D scenes? Are there any scalability issues?   - **Real-time Performance**: Can the method be applied in real-time applications, especially in robotics and embodied intelligence scenarios?   - **Ablation Studies**: Can you provide more detailed ablation studies to isolate the contributions of each component of the proposed method?   - **Generalization**: Have you tested the method on different types of 3D datasets to evaluate its generalization capabilities?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors did not provide an analysis of the limitations. For suggestions on improvement, please refer to the weaknesses section.

Response to the Rebuttal

Thanks for the authors' comprehensive rebuttal to my questions and sorry for the late reply. Their clarification on missing ablation studies and real-time performance analysis meet my satisfaction. Therefore, I will raise my rating to 6. However, I have to admit that I'm not an expert in this field, so please consider more about the review opinions of reviewers with higher confidence.