Summary
In this submission the authors propose a new pipeline that enables Large Language models to interact with trained object-centric NeRF models. They achieve this by utilising a pretrained meta-network that ingests the weights of a NeRF MLP and outputs a low-dimensional feature vector. This low-dimensional feature vector is then transformed to the language models input space via a projection layer similar to LLaVA. The method is trained on a dataset of 40K NeRF models which the authors extend with 240K paired text descriptions. Experiments on this dataset show favourable performance compared to prior works that solely work with images or point clouds.
Strengths
1. The proposed problem and method of interacting with NeRFs via a LLM is novel and very interesting, enabling potential new applications in robotics and AR.
2. The experiments performed in the submission show promising results, where the proposed method is achieving impressive results over the chosen baselines.
3. The dataset/benchmark and code will be released upon acceptance, which will make follow-up works and comparisons much easier.
4. The paper is very well written and concepts are easy to understand.
Weaknesses
In order of severity:
1. The comparisons that were performed in the experiments seem unfair towards the baselines. The baselines are not trained or fine-tuned on the dataset proposed in the submission and it is assumed that they would generalise on this new domain since they were trained on millions of images or hundreds of thousands of 3D shapes. Since the proposed dataset consists only of ShapeNet objects there will be a domain gap for the baselines to overcome, since they were trained on the Objaverse (Deitke, Matt, et al. "Objaverse: A universe of annotated 3d objects." CVPR 2023) or ModelNet40 (Wu, Zhirong, et al. "3d shapenets: A deep representation for volumetric shapes." CVPR 2015) Datasets in the case of the GPT4Point[58] and PointLLM[77]. The same logic applies to the BLIP2 and LLaVA baselines, which will not have seen many synthetic object centric images during their training. This weakens the overall argument to consider NeRFs as an input modality to LLMs because the presented comparisons are flawed.
2. While it is a valid argument that view selection is an issue for 2D-LLMs, as stated in line 123, NeRFs can render arbitrary viewpoints after training. It would therefore be possible to render multiple images from varying viewpoints and use all of them as input to modern Multimodal LLMs jointly, e.g. by concatenating their text-tokens after the projection layer. This would provide a more balanced comparison than the single-view baselines that are chosen in the submission, since a lot more information can be passed onto the LLM with multiple views. Another avenue to make a fair comparison to 2D-LLMs like LLaVA would be to encode 2D images in an MLP (as shown for example in: Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.) and then use these weights as input to the proposed method. Both of these points will probably be infeasible to address in a rebuttal but would be interesting experiments to support the use of implicit representations as input to LLMs.
3. It is a bit unclear from the description in line 215 if the test set contains only object classes not seen during training or if they were seen before. This should be clarified.
4. In table 5 it is not discussed what a ‘hard’ view is and what constitutes its ‘hardness/complicatedness’. This makes the table confusing and not self-explanatory.
5. Stating in line 90 that Ballerini et al. [5] are the first to utilise NeRFs as an input modality is not correct, what about NeSF (Vora, Suhani, et al. "Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes." arXiv preprint arXiv:2111.13260 (2021)) or Nerf-RPN (Hu, Benran, et al. "Nerf-rpn: A general framework for object detection in nerfs." CVPR 2023)? It probably refers to being the first to directly utilise NeRF MLP weights as an input modality when considering language tasks. This sentence should be re-written.
Questions
To focus the discussion about the weaknesses raised above here are some questions for the authors:
1. Please further justify not finetuning the baselines on the proposed dataset. How is this evaluation done in prior work and what justifications do they give for why they did or did not do this? Do these arguments also apply to this submission?
2. Is it possible to train a limited amount of NeRF models on Objaverse or ModelNet40 and then use the proposed pipeline without finetuning on them and compare to PointLLM and GPT4Point? This would be a very interesting experiment to showcase the generalisation ability of the proposed method.
3. Can you evaluate the 2D baselines with more views ? (related to Weakness 2)
Limitations
The authors addressed the technical limitations of their work quite well but limitations with regards to prior work are not discussed, as they do not exist in the current evaluation scheme.
Societal Impact is briefly discussed in the paper checklist.