Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

Summary

This paper proposes a 3D Multi-Label Language Model (MLLM) designed to perceive and represent 3D scenes at the object level. To interpret individual object instances, the authors develop object identifiers to convert the 3D scene into a series of distinguishable object tokens and present object-centric representations using foundation models. Experiments are conducted on various 3D scene-language tasks.

Strengths

1. Introducing Large Language Models (LLMs) into 3D perception and representation is a valuable and innovative research direction. 2. Leveraging foundation models to extract 3D and 2D embeddings shows significant potential for enhancing the performance and capabilities of the 3D MLLM model.

Weaknesses

1. The concept of object identifiers is not new, as similar methods have been previously introduced, such as in "Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V." 2. The authors claim the proposed model enables "efficient object referencing and grounding abilities," but this efficiency is not evaluated in the experiments. Furthermore, experiments on 3D referring expression segmentation are not provided, making this claim hard to substantiate. 3. It is unclear how the proposed object-centric representations address the problem of 3D data scarcity. 4. Table 2 lacks comparisons with several notable works, such as the state-of-the-art method CORE-3DVG (NeurIPS 2023) on the ScanRefer dataset. More existing methods should be included for a comprehensive evaluation.

Questions

Please refer to the weaknesses section.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

The authors have discussed the limitations and potential societal impact of the proposed method.

Summary

The paper proposes a new representation for 3D multimodal LLMs, a family of foundation models that repurpose LLMs to receive multimodal (visual and linguistic) input. Specifically, the paper advocates for an object-centric representation, where objects are first discovered (detected or segmented) with an off-the-shelf model, then they are fed as tokens to the LLM. This happens in the following way: i) objects are discovered in 3D using a 3D detector; ii) for every object we get a language identifier (e.g., <obj1>), a local point cloud and a 2D mask (by projecting the 3D mask back to multiple 2D views); the language identifier, featurized local point cloud and featurized 2D segment form tokens, that are fed to an LLM in the form of prompt. This formulation unifies many visual-language tasks as text generation. As a byproduct, using task-specific prompts, the model can be jointly trained on several tasks jointly, leading to improved performance. The results show quantitative gains on visual grounding, captioning and VQA.

Strengths

1. (Presentation) The paper is very clearly presented and easy to understand. The writing makes the right claims and all the useful details and questions are answered in the main paper. 2. (Contribution) The submission addresses an important problem, which is to ground the knowledge of LLMs in the visual world. The proposed scene tokenization is not novel per se, but its combination with MLLMs is a nice feature. Also, several important details, such as appropriate featurization using both multi-view 2D and 3D are very useful to see, because MLLMs haven't shown such good results so far using different input representation. 3. (Soundness) The results indeed show that the proposed approach manages to use MLLMs in an effective way. More importantly, the comparisons include baselines that also train multiple tasks simultaneously, so the gain doesn't seem to come more data alone. Although we cannot conclude that this architecture is better than baselines which may have trained on a single task only, it seems to be the strongest multi-task approach now.

Weaknesses

1. (Contribution) The paper enters the debate of the right representation for visual-language (VL) understanding without giving a clear answer, unfortunately due to current benchmarks' limitations. Object-centric transformers for VL tasks have been proposed long ago (see VilBERT, VisualBERT, LXMERT, OSCAR, UNITER etc for 2D VL understanding), but then got superseded by one-stage approaches like MDETR. The issue with two-stage object-centric models is the definition of a vocabulary. Open-vocabulary methods cannot really rely on detector bottlenecks, since they, by design, have a limited vocabulary. We cannot possibly enumerate all the concepts a user may refer to. That said, in 3D VL understanding and especially grounding, most approaches are indeed object-bottlenecked. This can be attributed to ScanNet being the base dataset for most 3D VL benchmarks. ScanNet limits the scope of in-domain approaches to few classes. One cannot for example detect the legs of a chair using ScanNet, since parts of objects are not annotated. This is true for both one-stage and two-stage approaches, as long as they are trained on ScanNet. But among the two directions, the one more promising to extrapolate on a broad domain seems to be the non-bottlenecked one. However, the object-centric tokenization is not bad, it provides a nice abstraction of the scene. My concern is that a detector will not cover some useful parts of the scene and, as a result, that part of the scene won't be visible at all to the model. This can be due to imperfect prediction or even limited concept vocabulary. Given how impressive the VLMs' generalization is, limiting them with in-domain detectors may be handicapping them. 2. (Soundness) It seems that a lot of quantitative gain comes from using 2D features. While it is fair to use 2D features in the proposed approach, I'm not sure whether the main factor of good results is the use of object identifiers, 2D features or multi-tasking through the unification of the output space. For example, an approach that does the same unification (with different prompts of course) and uses scene tokens and multi-view features (for example by unprojecting 2D features from multiple views and performing some voxelization) would be the best baseline to validate the importance of object identifiers. Which baseline or ablation represents that direction? Right now, we can mainly conclude that 2D features are very important and that the format of object identifiers is better than other alternatives, but we cannot safely conclude that object identifiers are the most useful component. Another interesting fact that I noticed in this paper is that the best-performing models on grounding (ScanRefer) are the ones trained only on ScanRefer. In fact, the proposed approach is the only multi-task approach that beats those models. I believe it would be useful to also see some results on Nr3D and Sr3D for several reasons. First, the margins on grounding are a bit narrow, so we don't know whether the most competitive approaches lack due to less data or architecture. Evaluating on more datasets could provide some more evidence. Second, Nr3D and Sr3D use ground-truth boxes. It's good to see what is the limit of the proposed approach if a part of perception is perfect.

Questions

Overall, I appreciate the presentation of the paper, its claims, good results and interesting technical contributions. At the same time, thinking broader of the field, I'm not sure whether this paper provides enough evidence towards the right direction; going through an in-domain bottleneck may drastically limit the generality modern LLMs/VLMs can offer. Moreover, the ablations lack some targeted experiments that would help us verify the proposed components. I would appreciate if the response can clarify/add some of the following: * Discussion on the scope of using object-bottlenecked 3D VL models for the general field (beyond evaluating on ScanNet). * Discussion on the relation of this work to previous object-centric tokenization approaches from the 2D VL domain. * Adding results on Nr3D/Sr3D. * Adding some more tergated ablations that switch off object identifiers. I will adapt my score after the discussion. --------------------------------------------------- Post-rebuttal, some concerns are addressed and I'm increasing my score to from 4 to 6

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Addressed

Summary

This paper proposes a 3D MLLM that can understand the 3D environment at the object level. The proposed work designs object identifiers that are projected into language token space and can be understood by LLMs, which unifies several 3D scene understanding tasks into the same format. These object identifiers give a more natural way for humans and MLLM to reference specific objects of interest in a 3D scene. The paper extracts features from 3D scenes using several 2D and 3D pretrained models and finetunes the MLLM on 3D scene understanding tasks. Experiments were conducted on several benchmarks based on ScanNet and the results show the proposed method achieves state-of-the-art performance, surpassing prior arts by a large margin.

Strengths

1. The paper decomposes a 3D scene into objects and develops object tokens to represent them for interaction with LLMs. This is a more natural way for people and LLMs to refer to object entities in the scenes. 2. The paper conducts extensive experiments across several 3D scene understanding tasks (Grounding, VQA, and Captioning), showing the versatility of the proposed method. The reported results show that the proposed method achieves state-of-the-art performance and outperforms the existing expert and LLm-based models by large margins. 3. The paper also conducts a study on video input using a 2D video instance tracking model, showing that the proposed method can still work without the 3D model. 4. The paper is well-written and the code is provided in the supplementary material.

Weaknesses

1. The performance of the proposed method is highly reliant on the ability of the pretrained 2D and 3D models (Mask3D, DINOv2, DEVA). Specifically, since the scene is represented as discrete object tokens, some fine-grained information existing in the full 3D scenes is lost. For example, if the 3D segmentation model wrongly merges 2 chairs into one, there is no way for the proposed method to recover. In this regard, I encourage the authors to provide some case studies about how the model fails (or succeeds) in this way. 2. Although multiple tasks are tested, they are all based on the same underlying dataset, ScanNet. Thus it's unclear how the proposed method works on other datasets. While this is maybe because the ScanNet has most established benchmarks on these 3D scene understanding tasks, I still want to see how the proposed method works on other datasets (e.g. some outdoor datasets from AV literature, or simply the next iteration of ScanNet, ScanNet++). Such evaluation can be quantitative or qualitative. It's also interesting to see how the model trained on ScanNet only can transfer to other datasets. 3. I'm interested to see the zero-shot, open-vocabulary generalization of the proposed method. As the proposed method is based on several foundation models, it probably has the ability in such settings. I won't consider this as a major weakness though, as this is not claimed as a contribution in the paper. Minor issues, typos, and grammar errors: * Ln 156, "due to due to".

Questions

1. Ln 183 mentions that DINOv2 has superior handling of local features within images. Why and how does this matter in the 3D scene understanding task? How does it influence the final performance? The relevant ablations study is currently lacking.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors provide a discussion on limitations and societal impact in the appendix. Yet they don't provide qualitative examples or discussion on the failure cases, which I highlighted in the weakness section.

Summary

This paper aims to enhance the efficiency in interpreting individual object instances and improve referencing and grounding capabilities for intricate scene comprehension. The method decomposes the input 3D scene into object identifier tokens. Experimental results on 3D scene-language datasets demonstrate the effectiveness of the proposed approach.

Strengths

1. The paper is well-written and easy to follow. 2. The experimental results on 3D scene-language datasets shows the performance improvement and effectiveness of the proposed method. 3. The training schema of the model is single-stage yet effective on downstream task.

Weaknesses

1． The authors do not provide the details about detectors, encoders, multi-modal inputs and LLMs used in other methods in the Table 1. These designs has already built a high-performance baseline, for example, the baseline results in Table 2 have exceeded other methods in the Table 1. It is difficult to judge the fairness of the comparisons in Table 1. 2． This paper lacks the performance on ScanQA test set. 3． The ablations are insufficient. The paper lacks the ablations about different size of LLM, training and fine-tuning schema. 4． The paper lacks the computation and time cost about the proposed method. 5． This paper does not provide sufficient discussion with object-centric representation learning methods on 3D vision-language. For example,[a] explored the object-centric representation learning with contrastive learning. And the object-level tokens are used in LEO[22], 3D-LLM[20], etc. [a] Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding, AAAI2024

Questions

1. More ablations to show the effectiveness of object identifier, such as the order of identifiers, only combining a 3D object token embedding and a 2D object token embedding for each object without the specifical token for identifier. 2. More ablations about the LLM used for 3D-VL learning, such as different types of LLMs , different sizes of LLM, multimodal large models such as LLAVA[b] or other 3DLLM[22] 3.The details about the training set, such as the scale of 3D, 2D and language, the statistics of identifiers . 4.The details about the inputs at inference, like will all objects of a scene be combined as input, and how to combine? [b] Visual Instruction Tuning, NeurIPs2023

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors provide the limitations about the reliance on pre-trained models, data scarcity, and the experience hallucinations.

Thank you for your response

I would like to thank the authors for their effort in the rebuttal. My main concerns were regarding the object bottleneck, references to related work, additional evaluations, more targeted ablations. * Discussion regarding the object bottleneck: while we don't expect a single paper to solve the debate, this paper positions in favor of two-stage approaches for limited data setups. That opens indeed a new discussion given recent advances in mixed 2D-3D training [1] but I agree that it is an avenue for future work. * Additional connections to related work were added. * Additional evaluations on ReferIt3D were added. * One additional ablation was added, but I'm not sure I get all the details. Are encoded bounding boxes fed to the LLM? Some description of this baseline would be helpful. If that's the case, one related approach is LanguageRefer [2]. I was also imagining that scene tokens (after some voxelization) could be fed directly to the transformer. It would also be interesting to ablate the effect of multi-tasking. Is the architecture strong on its own or it benefits a lot from the unification of the output space? For that, some single-task results would be useful, e.g. on grounding. For the final version, please consider adding the additional discussions on the broader position (two-stage vs one-stage), related work and additional results and ablations. If possible, including single-task results would provide a great insight on disentangling the architecture design and output unification. While the last is studied for NLP, it's not well-studied for 3D VL understanding. Under these conditions, I'm increasing my score from 4 to 6. [1] ODIN: A Single Model for 2D and 3D Segmentation, 2024 [2] LanguageRefer: Spatial-Language Model for 3D Visual Grounding, 2021

Response to rebuttal

Thank you for providing the rebuttal. However, I noticed that the performance of the proposed method on hard objects is subpar, as shown in Table A. Additionally, the comparison involving pre-trained 2D and 3D encoders—compared to methods without pre-trained encoders like CORE-3DVG—raises some questions, particularly since the performance is even lower than CORE-3DVG on Acc@0.25. Therefore, I will revise my rating to a borderline reject, but I remain inclined towards rejection.

Thank you for your efforts

We sincerely thank the reviewers and AC for their efforts during the review and discussion process. We would like to briefly highlight the motivation and contributions of our work. The trend in the 3D field is moving towards generalist models, yet existing methods struggle with effective object referencing and grounding within LLMs. **Our work is the first to address this challenge and unify 3D tasks—grounding, captioning, and QA—within a single LLM-based model, without the need for task-specific designs or fine-tuning.** Our goal is not to claim that the proposed model is completely superior than all specialist models in their specific tasks or metrics. We do not deny that specialists may continue to improve with data scaling or better pre-trained weights in the future, but their task-oriented designs inherently limit them to specific tasks. **Compared to both specialist and LLM-based models, the superior performance of our model across all benchmarks establishes a solid baseline for the future development of 3D generalist models.**