A Unified Framework for 3D Scene Understanding

Summary

The paper proposes a unified 3D segmentation frame work for six 3D segmentation tasks. It enhance the performance through building the inter-task connections. The model could achieve the state-of-the-art performance in individual tasks even comparing to the models specialized for individual tasks.

Strengths

The proposed model unifies 6 different 3D segmentation task in one framework, with no modules specialized for specific task. The authors build explicit inter-task relations to further improve the performance of individual tasks. Detailed experiments and comparison are reported.

Weaknesses

1. The writing needs improvement. Some notation and writing flow are confusing. For example, K_v in line 155 is not introduced. The multiplication in equation (3) is not clear (is it a matrix multiplication?). Positive sample and negative samples are not introduced in section 3.2 (line 206). mask_pos in equation (6) is not introduced. 2. No explanation on how to compare models/design choice when the evaluation metrics for different tasks are not consistent. For example, if model A is better in interactive segmentation while model B is better in referring segmentation, how to compare? 3. Without the extra fine-tunning trick, the proposed model has a worse performance over previous SOTA OneFormer3D [16] on SS and IS tasks (table 2). According to Table I and IV in appendix, the performance of the proposed method is questionable. 4. The authors empirically find the interactive segmentation is the best task for mask predictions (line 57-58) but no analysis on the reason. Only one value is reported in table 1, which could not validate interactive segmentation has superior performance.

Questions

1. As is mentioned in the weaknesses, how do you compare models/design choice when the evaluation metrics for different tasks are not consistent? For example in table 4(a), how would you compare line 2 and line 4? Which one is better? 2. The design choice of Mask Decoder Layer is weird. Why do you choose to do a cross-attention first then followed by a self-attention? Also, is there duplicate information when doing the cross-attention between q and f_i, since q_u in q is obtained from f_i? Why not do the self-attention for q_u and cross-attention with q_p and q_t? 3. Table 6 shows that unifying the tasks are hurting the performance, then what is the point of unifying all the tasks together? Adding new tasks (OVS, Referring, Interactive) is hurting PS, SS, IS. Would these new tasks help PS, SS, IS? 4. As is mentioned in the weaknesses, table 1 could not support the claim that interactive segmentation is superior in mask prediction. Also, is there any intuition why this specific segmentation task is superior?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The paper addresses the limitation and claims there is no societal impact.

Summary

This work proposes UniSeg3D, a framework to unify 3D point cloud segmentation tasks. Compared to previous work that unifies 3 tasks, UniSeg3D additionally incorporates interactive segmentation, text-referring segmentation, and open-vocabulary segmentation. In total, six tasks are unified in a single Transformer decoder architecture, and techniques such as knowledge distillation, contrastive learning, and two-stage fine-tuning are applied to boost performance. Consequently, UniSeg3D achieves comparable or superior performance against existing state-of-the-art baselines.

Strengths

1. Novel architecture for incorporating six segmentation tasks under one single model, which appears to be effective and flexible for extending to additional tasks. 2. Solid experiments. The authors conducted experiments on 3 datasets and various ablation studies to demonstrate the effectiveness of the proposed method.

Weaknesses

1. Marginal performance gain. While referring segmentation and interactive segmentation tasks enjoy noticeable improvements in UniSeg3D, the performance of generic segmentation tasks seems to be on par or worse than training alone (see Table 1). This led to questions about the motivation for the proposed unification. Further justifications, such as the standard deviation of the performances, could make the results more convincing. 2. Additionally, in Table 6, unifying the additional 3 tasks turns out to hurt the performance of generic segmentation tasks. How do the authors justify their motivation? 3. Some minor issues in writing and notations. See the Questions below.

Questions

- Equation 3: The current notation is for cross products while it looks like dot products are intended. - Table 1: where does the Table 1 come from? - Notations: $e^P$ and $e^T$ in Equation 4 are not the same as the $e$'s in Figure 2 and there are no corresponding notations in Figure 3. Consider using a different notation or adding corresponding legends in Figures 2 and 3 for better clarity. - Section 3.3: the motivation of inter-task association is not clearly stated, how exactly can the tasks benefit each other? For example, how can referring segmentation benefit interactive segmentation?

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have discussed their limitations in Section 5. Given the paper is titled A Unified Framework for 3D Scene Understanding, the reviewer would like to point out that point cloud segmentation is one aspect of 3D scene understanding, so it would also be worth discussing how such a unified segmentation framework can potentially help other 3D understanding tasks.

Summary

This paper proposes UniSeg3D, a unified framework for six 3D segmentation tasks that achieves SOTA results on the six tasks. The authors propose to use knowledge distillation and ranking-based contrastive learning to enhance inter-task knowledge sharing and the overall performance. Extensive experiments are done to prove that UniSeg3D is powerful. Comprehensive ablation studies are performed to prove the effectiveness of the design.

Strengths

1. UniSeg3D is the first framework that unifies six tasks in 3D segmentation, and the comprehensive experimental results prove the effectiveness of the design. 2. Interactive segmentation-guided training is insightful. Analysis of its impact on inter-task formulation is comprehensive, enlightening future directions. 3. Well-written manuscripts with illustrative figures.

Weaknesses

1. Since the feature for visual prompt is sampled from the superpoints, the quality of the visual prompt significantly influence the overall performance of the model. 2. The experiments are only conducted on ScanNet-based datasets, as a unified model, the authors should provide more experiments on different datasets to validate the method as done in previous works[1][2]. 3. Unlike previous works[2][3], UniSeg3D directly learn the relation between text and 3D without any 2D supervision or guidance. It is a concern that how "open" is this framework in OVS task. Some visualized experiments on open-set text queries as in [2][3][4] could answer the question. [1] Zhu, Ziyu, et al. "3d-vista: Pre-trained transformer for 3d vision and text alignment." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [2] Nguyen, Phuc, et al. "Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Takmaz, Ayça, et al. "Openmask3d: Open-vocabulary 3d instance segmentation." arXiv preprint arXiv:2306.13631 (2023). [4] Peng, Songyou, et al. "Openscene: 3d scene understanding with open vocabularies." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Questions

Please refer to the weaknesses.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

For the first time, this work proposes a unified model for several point cloud segmentation tasks, including panoptic, semantic, instance, OV, interactive, and referring segmentation. This work uses the typical query-based transformer perception architecture with the proposed knowledge distillation losses for superior performance, outperforming previous methods, which shows the effectiveness of the designed model.

Strengths

1. This work addresses an important problem of 3D scene perception with a unified model and achieves great results. 2. The proposed model is simple and effective. 3. The overall writing is fluent and clear.

Weaknesses

1. The main weakness of this work is that the proposed architecture is widely used for multi-modal perception tasks, limiting the model’s novelty. However, this is acceptable as long as the model’s performance is indeed great, as simple and effective models are usually similar. 2. The paper claims knowledge distillation losses as one of its contributions. However, according to the ablations, adding these losses only makes marginal changes. Still, I think this work has a solid contribution to the field (if open-sourcing the codes with reproducible results) and should be accepted.

Questions

Some implementations of the method are unclear. 1. The detailed process of producing super points. 2. The criteria for sampling the corresponding points from the clicks. 3. For the referring segmentation, how to decide the target segmentation mask from many text tokens? 4. For L182-186, I am confused that isn't formula-5 already satisfied by using formula-4?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

See above.

We are open to any further discussion.

Dear Reviewers, We sincerely appreciate your time and effort in reviewing our paper. We hope our explanations have addressed your concerns. As we are in the discussion phase, we welcome any additional comments or questions regarding our response or the main paper. If further clarification is needed, please do not hesitate to mention it, and we will promptly address your inquiries. We look forward to receiving your feedback. Best regard, Paper 1782 Authors

Thank you for your response

Thank you for your detailed responses, which have made the work more comprehensive and addressed my concerns regarding the open-vocabulary attribute. I hope the author can add the additional visualization results to the revised version of the paper. Considering the impact of this work, I'll maintain my initial score.