Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs. Code is available at \url{https://github.com/Kunhao-Liu/3D-OVS}.
Paper
Similar papers
Peer review
Summary
This paper proposes a new approach for fusing in 3D the output of 2D neural networks (CLIP an DINO) applied to the analysis of multiple views of a given static scene. The main application is open-vocabulary 3D segmentation of radiance fields. Compare to trivially fusing CLIP features, the method introduces a few improvements, such as re-normalising the response of CLIP and using DINO to regularise the segments. These improvements are shown to help segmentation significantly. Comparison to recent works, including concurrent work such as LERF, are included.
Strengths
* Fusing open-ended features such as CLIP in 3D can be useful in applications, including, as demonstrated here, for open-vocabulary 3D segmentation. * I appreciated the effort to compare to very recent *concurrent* work like LERF. The fact that the method outperforms the latter, at least in the authors' experiments, is a further bonus. * There are a few ideas (class re-balancing, multi-scale feature selection, relevance renormalisation and consistency with DINO features) which are shown to improve significantly the performance compared to simply fusing CLIP features. This illustrates some difficulty with the latter approach, and also shows simple fixes to these issues.
Weaknesses
* The idea of fusing 2D features in 3D reconstruction using neural fields is not particularly novel as such. Semantic NeRF has done it for segmentation, N3F and FFD for unsupervised features like DINO, Panoptic NeRF and others for semantic segmentation, etc. FFD already included some early experiment with open-ended language-based features (and this paper compares to them). * The math needs significant revision. Some issues amounts to typos and inaccuracies, but others really make it hard to understand how the algorithm works in detail. Some aspects of the algorithm are also slightly surprising and need some discussion. See below: * Eq. (3) is the main loss that allows to learn the radiance field model, including its extension to predict the multi-scale CLIP features F and the scale selection vectors S. I understand most of this *except* why and how this can learn feature selection. First, the equation does not make literal sense. The inner product $\langle \hat F(\mathbf{r}) \cdot S(\mathbf{r}) F(\mathbf{r} \rangle$ misses a comma, but I expect that the intended read is $\langle \hat F(\mathbf{r}) \cdot S(\mathbf{r}), ~ F(\mathbf{r} \rangle$. Second, the $\cdot$ product is unclear. $F$ is a $N_s \times D$ matrix and $S$ is a $N_s$-dimensional vector. Is this an element-wise product with broadcast? Thirdly, $\hat F$ are the multi-scale CLIP features "fused" (i.e., rendered) by the radiance field and $F$ the ones predicted from the 2D image. Why comparing these two weighed by $S$ should result in $S$ learning to select the appropriate scale of CLIP features to use for that point? * Eq. (6): Is this meant to be $\langle T, F_I \rangle$? The notation $\langle T \cdot F_I\rangle$ does not make literal sense. Furthermore, $T$ is a $C\times D$ matrix and $F_I$ is a $N_s\times D\times H\times W$ matrix. The output is $C\times H \times W$, so we can sort of guess what the authors mean to do, but the notation does not show how the scale dimension $N_s$ is reduced (presumably using the scale selector $S$, but the notation does not reflect that). * The RDA loss of Eq. (8) is a bit problematic. First, from a modelling perspective, it seems to be based on spatially re-normalizing the response of the CLIP features to all possible classes $c$ an a class-by-class basis (see also Fig. 2). This is done to prevent certain classes from "swamping" others. However, what happens if a class is *not* present at all in a scene? Wouldn't this *force* the class to still be segmented, thus incorrectly? Second, there are issues with Eq. (8) as such. The quantity $\bar R(\mathbf{r})$ is a $C$-dimensional non-negative vector, but, unless this is done implicitly, it does *not* sum to one. I.e., its is not a *probability* vector. Can you really interrupted Eq. (8) as a sum of JS divergences? * Eq. (9) what is the notation $\cos f_{hw} \cdot f_{ij}$? Do you mean $\langle \frac{f_{hw}}{\|f_{hw}\|},\frac{f_{ij}}{\|f_{ij}\|} \rangle$ ? * What is $z$ in Eq. (9)? * I understand that the authors are trying to implement the segmentation distillation idea of [49] -- Eq. (12) certainly bears resemblance to that work. However, I had a really hard time understanding the notation used in Section 3.3 overall. * The evaluation is only carried out on 10 custom scenes with comparatively little clutter. Given that the authors have been willing to compare to LERF (which is concurrent work, so this is definitely not required) I wonder if it would make sense to run the assessment on *their* 13 custom scenes as well.
Questions
* See the technical issues above. Can you clarify the notation, and, especially, the question about renormalisation in the RDA loss? * In the scaling experiment of Table 2, bottom 4 rows, you use a very small number of views (according to the text, just 2-3 -- se line 279) are sufficient to obtain good segmentation performance. Do you mean that you are still using all *RGB* views for 3D reconstruction and just 2-3 views with CLIP features, or that you are using only 2-3 features in total, including for 3D reconstruction? The latter would be very surprising.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
There is no substantive discussion of limitations (e.g., no failure cases demonstrated, no section or paragraph discussing limitations). Note that the NeurIPS guidelines encourage explicitly the discussion of limitations. There is no discussion of societal impact, ethics, copyright, data protection etc. I don't believe that any of this is particularly relevant to this paper though. I do not see direct potential harm stemming from this research, and the datasets used appear custom-collected and do not contain personal data.
Summary
This paper introduces a method that incorporates pre-trained CLIP features into Nerf for 3D open-vocabulary segmentation. The authors also utilize pre-trained DINO features to regularize the segmentation process and achieve precise segmentation boundaries. Additionally, they propose a Relevancy-Distribution Alignment loss to address ambiguities in CLIP features.
Strengths
1. The paper demonstrates a fluent and well-structured organization. 2. The experimental results exhibit favorable performance.
Weaknesses
1. The testing data is relatively simple, consisting of views with limited perspectives, and does not include results on larger and more complete 3D scenes (e.g., the benchmark proposed by Lerf). 2. Regarding the Feature-Distribution Alignment issue: If there are multiple objects of the same class, such as x1, x2, x3, in an image, their DINO features' similarity may be lower than b, but the Feature-Distribution Alignment could make them assigned to different classes, which is unreasonable. Although λ_neg is small, the supplementary material demonstrates the criticality of neg_F, and removing it significantly deteriorates the results.
Questions
1. How would the performance be affected if the Relevancy-Distribution Alignment was not used during training and instead the segmentation probabilities were spatially normalized during inference? 2. How does the proposed method perform on Lerf's benchmark?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The processing of multi-spatial features of CLIP incurs substantial computational and storage resource requirements.
Summary
This paper tackles the task of 3D scene segmentation using CLIP and DINO distilled into a NeRF representation. Since CLIP and DINO are pre-trained, the method works in an unsupervised fashion i.e., no additional per-point training labels are needed. This is achieved by applying CLIP to small crops in a sliding window fashion resulting in pixel-aligned features. This is repeated for multiple scales, resulting in multiple pixel-aligned CLIP features maps, one for each scale. The paper introduces two new losses (FDA and RDA) that rely on a closed-set of classes during training of the NeRF. The method is evaluated on their own dataset and annotations, and compares to multiple, recent methods (LSeg, ODISE, OVSeg, LERF, FFD).
Strengths
A novel “volume-selection” scheme used to align per-image CLIP features to per-pixel CLIP features, in addition to two new losses: Relevancy Distribution Alignment loss and Feature Distribution Alignment loss that do not require a dedicated pixel-aligned CLIP method.
Weaknesses
I struggle a bit with the intermixing of the term “open-vocabulary” and the fact that the method relies on a known / closed set of classes C. E.g., “open-vocabulary text description for each class” (l.119) somehow implies that the set of classes that we want to segment is known a priori, whereas open-vocabulary implies that there are no such restrictions. Importantly, the two losses presented in Sec.3.2 and Sec.3.3 seem to rely on a predefined set of classes C - i.e. **each NeRF is actually optimized for a closed-set of classes, and if the classes change, the NeRF needs to be retrained**. The method can be seen as open-world in the sense that there is no inherent limitation to the classes, however an optimized NeRF is a closed-set representation. Is that correct? I think this aspect should be discussed in more detail in the limitations. The writing could be improved, numerous aspects are still unclear and should be properly introduced and explained (Please see the minor points under Questions - I consider them minor since it can be improved in an updated version, however there are so many points that are unclear that the overall presentation of the method is limitted). This paper follows an unfortunate trend that is observable across the field of LLM-based methods for open-set scene understanding: each paper presents a new method and at the same time introduces a new (relatively small) evaluation dataset on which they improve over prior/concurrent work. It is clear that such evaluations are not entirely convincing since the results could be cherry picked and favoring the newly proposed model. This paper is unfortunately no exception to that trend. In particular, the dataset selection is motivated by the fact that ScanNet [5] is too limited in terms of classes - however there was a recent extension to ScanNet200 that should mitigate this problem. Also Replica [70] does feature 50-80 semantic classes (depending on the scenes) which shows a long-tail distribution. Hence, there are existing datasets (maybe not perfect) but at least established ones and certainly not worse than very few newly recorded scenes. Further, since the proposed method compares with LERF, it could also compare on the LERF dataset. The argumentation / motivation in l.28-31 is unfortunate: it argues that the major challenge of open-set segmentation is the lack of datasets, however that is not really the case since open-set datasets cannot exist as it would require an infinite amount of all potentially existing labels. The contributions (1) (in line 71-73) - how does it differ from Decomposing Nerf [4] ? They also use CLIP and DINO in a very similar fashion. Instead of LSeg one could use the proposed multi-scale CLIP feature maps and then compare, this would be a reasonable baseline.
Questions
How is the selection vector S_x obtained? Eq. (2) shows that we can render it from the optimized NeRF, that is clear. However, we also need it already during the optimization, see Eq. (3). How does that work, i.e., how can the value S(r) that is only obtained via rendering of the optimized NeRF at the same time be used to optimize the NeRF? Finally, I do not see an ablation study that shows the benefits of the 3D Selection Volume. Does the 3D selection volume improve over simply averaging multi-scale features as described in l.151? Minor: - l.4 “lack of [...] open-vocabulary segmentation datasets” I would even argue that open-vocabulary datasets are inherently impossible as it would require exhaustive labeling of all possible queries. - l.7 ”finetuned with close-vocabulary datasets” this is only the case for a subset of visual-language models such as LSeg, other approaches are not fine tuned on closed sets of data e.g. OpenSeg. - l.35 same point as before: OpenSeg is not fine tuned on a closed set of classes but weakly supervised via image captions. - l.46 - ‘ambiguous similarities with text description’ what does that mean? - l.53-55 I don’t understand this sentence, can you break it down for me? What are CLIP feature ambiguities? What is a segmentation probability distribution? What are class relevancies? Until l.62 is similarly unclear - I think it’s mostly unclear because the used terminology is not properly introduced / defined. I can guess what is trying to be achieved but that is not my job, it should be clearly written. - l.126 “ambiguous similarities” - what does that mean? - l.127 what are “relevancy values”? Fig.2 is not that helpful since it also does not explain the terms. Again, what is a segmentation probability distribution (l.130)? I can sort of guess it (maybe I’m wrong, maybe I’m right?), why not clearly define and explain these terms since a large part of the main contribution depends on them? - l.169 Still unclear meaning of “ambiguities of the CLIP features”
Rating
2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
1 poor
Presentation
3 good
Contribution
2 fair
Limitations
Limitations are not addressed. For example, one could discuss the impact of the closed-set of classes C required to optimize the NeRF since this has implications on the practical application (see under weaknesses).
Summary
This work addresses the open vocabulary segmentation by leveraging pretrained foundation model, DINO and CLIP. They firstly lifts 2D CLIP features into NeRF. Then, relevancy distribution alignment loss is used to mitigate the CLIP features's ambiguity. Feature distribution alignment loss is used to improve object boundaries from DINO feature.
Strengths
3D open vocabulary segmentation is an interesting and impactful topic. It is very challenging to use a fully supervised way for training such a model since annotation costs large amount of human labor. This paper leverages 2D foundation models. The presentation is fairly good and the proposed relevancy distribution alignment and feature distribution alignment losses are well motivated.
Weaknesses
There are many unclear parts that make me feel hesitated to suggest acceptance of this paper. 1. how they used CLIP is not well justified. Will the patch-wise method break the CLIP feature space? I assume many patches covers only background or small portion of objects. 2. Why DINO? CLIP and DINO are two most important models for this work. So, motivating the utilization of them is important. Especially for DINO, why it is good for object boundaries. Visualization would help or ablation study will help. 3. the most concerning part is the experiments. The method is validated in limited settings. The class set is small that is not enough to show "open vocabulary" capability. The dataset is mall.
Questions
See weaknesses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The unclear parts in the paper and the limited experiments jointly make the performance of this work not convincing enough.
Summary
This paper trained a NeRF model leveraging on two self-supervised trained models CLIP and DINO. By distillating multi-view image features from these two large models, the NeRF model is able to synthesize accurate object segmentation masks on unseen scenes with long-tailed distributions. Experiments on 10 self-collected scenes shows that high-quality segmentation masks can be predicted after per-scene training without explicit segmentation annotations.
Strengths
- This paper proposes a easy-to-follow pipeline to reach accurate open-world 3D segmentation by taking advantages of two pretrained large models CLIP and DINO without access to segmentation ground truth masks. - The experiments on self-collected scenes prove the capability of proposed method, and the ablations cover most of key design choices made in the paper, with adequate implementation details.
Weaknesses
- Limited experiments on popular benchmarks. My main concern lies on the limited evaluation as 'open world 3d segmentation' is a challenging topic. - In this paper, 10 scenes with multi-view high-res images are taken for experiments. However, more diverse results on different datasets are expected to really prove the capability of ‘open-world 3D segmentation’. Existing datasets like ScanNet, VG150 still contains imbalanced long-tail distributions. How the proposed methods work on ScanNet, for example, where captured images are in VGA resolution and lower quality compared to used data in the paper. - There are many hyper-parameters need to be considered during training. Are they sensitive to difference scenes/datasets? Otherwise, the overall experiments may look too curated to prove the main statements. - One question to ask is whether the model sacrifice the ‘close-world’ capability to complement the open-world one. Therefore, it would be good to see how trained methods work on different categories with different types of text prompts. - What is the impact of text descriptions? Are the given text carefully chosen or some general text input covering key information are good enough for generating promising labels? Other comments: - Why only 6 of 10 scenes are shown for numerical experiments with baseline methods and 4 scenes are used for ablations? I see a complete 10 scene results of proposed methods in the supplement but not of baselines. - One question rather than weakness. Could authors explain why to choose to learn CLIP features with a shared TensorRF volume with colour instead of separately. Is there any specific reason or ablations for such design choice?
Questions
Please see weaknesses above where my main concern is the lack of analysis on diverse popular real-world scenes to really prove the model's capability. Overall this paper is attempting to tackle a challenging task and I think more discussion could better strengthen the contributions.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Limitations and potential negative societal impacts have been discussed in the supplement.
Response to rebuttal
I thank the authors for clarifying their maths in the answer. I think I can now understand the details in a more rigorous manner, but their comment "Because there are lots of details which can be better explained with code, we thus use such a simple notion here" in the rebuttal leaves me perplexed. I understand the desire to simplify the notation, but using a notation which is suggestive but factually incorrect does not seem to me a great stylistic choice. It is possible to use a simple notation while carefully explaining its meaning to ensure that it isa also correct. After all, it took them just a few lines in the rebuttal to clarify this. The new evaluations, including comparison with LERF, are interesting -- and also go beyond what is required given that LERF is concurrent work. Overall, my inclination is to accept the paper.
Regarding the comparisons with (concurrent) LERF, and since I think it is not very clear from the paper and also marginalized in the rebuttal, I want to highlight again that the proposed methods relies on known object-classes for training the NeRF. This is not the case for LERF (and other baselines). I see two potential problems: - the comparison with LERF (and others) seems unfair as LERF does not make use of known object classes. - the method in fact cannot be called open-vocabulary since it relies on knowing the closed-set of object classes to optimize the NeRF. I am curious about your opinion on these points.
Thanks for your reply. Here we address your concern in the following three aspects: 1. **Our method significantly differs from close-vocabulary segmentation methods.** Close-vocabulary segmentation methods can only segment labels from a pre-defined set of classes in a dataset. When handling new labels, they require well-annotated training data at image, region, or pixel-level depending on the new labels [1]. Our method requires only the new labels, and it does not require any annotations of the training data (similar to applying CLIP under the zero-shot setting). It’s worth noting that, unlike 2D visual recognition tasks, there is no clear definition of open vocabulary in 3D counterpart due to the lack of large-scale 3D foundation models. The typical approach distills knowledge from 2D foundation models towards 3D tasks, in which there is no clear norm to avoid using text labels during the distillation. 2. **Comparison with LERF[2].** Under the context of 3D open-vocabulary segmentation, LERF and our method require the same inputs for generating segmentation: pre-trained CLIP, DINO, multi-view images, and user-provided text labels. The user-provided text labels are also used for the same purpose – distilling knowledge from the CLIP. They are just used at different stages – LERF uses them at the inference stage and hence it does not acquire sufficient semantic knowledge for segmenting object boundaries accurately. The comparison is thus fair though the implementations of the two methods are not completely the same. 3. **Comparison with the baselines Sem(ODISE)[3,4] and Sem(OV-Seg)[3,5].** We would highlight that our setup is identical to that of the two baselines which utilize user-provided text labels in the same manner as our method. Both baselines necessitate the target text labels to acquire 2D segmentation maps before NeRF training, and they are both considered open-vocabulary segmentation methods. [1] Panoptic Lifting for 3D Scene Understanding with Neural Fields [2] LERF: Language Embedded Radiance Fields [3] In-Place Scene Labelling and Understanding with Implicit Scene Representation. [4] Open-vocabulary semantic segmentation with mask-adapted clip [5] Open vocabulary panoptic segmentation with text-to-image diffusion models.
1. I only partially agree with this point. It is correct that closed-vocabulary methods can only segment labels from a pre-defined set of classes, but the same is true for the proposed method: The proposed losses **L_RDA and L_FDA can only be computed if the (closed) set of object classes is known** prior to optimizing the NeRF, cf. "method requires only the new labels" i.e. it *does* require training labels. As soon as a method requires a closed set of labels during training, we can no longer talk about open-set, specifically because it is no longer possible to query arbitrary objects from the trained scene representation during inference, thus **it is not an open-set method**, the trained NeRF is tailored to a closed-set of classes (and needs re-training for a different set of classes). However, I see that the method does not require annotated per-point labels, so **the method can be called weakly-supervised, or unsupervised**. 2. The claim that LERF and the proposed method require the same inputs is not accurate and potentially missleading: During training of the NeRF (generating the 3D scene representation), **the proposed method additionally requires a closed set of object classes during training**, i.e., the method only works if the object classes in the scene are known. This is not the case for LERF. The claim that the class labels *``are just used at different stages"* is exaclty the point: *it is somewhat similar to using test labels during training*. In particular it gives an unfair advantage to the proposed method which solves a much easier task of segmenting a scene into a known set of classes. However, methods like LERF are more general, they compute an open-set 3D scene representation during training, that can be queried for arbitrary objects during inference. The proposed method however would require retraining every time the queries change. 3. Comparison with ODISE and OV-Seg. *"Our setup is identical to the two baselines"* - This is incorrect: In Sec. 3.6. of ODISE, it is stated that *"the test categories may be different from the training ones"*. This is not the case for this approach since each scene is optimized during training for the test labels of that particular scene (via the L_RDA, and L_FDA). There is also no experiment that would demonstrate that the method is able to segment classes not seen during training. In particular, an important experiment is missing that would potentially highlight this point: How would the proposed method perform on an unseen set of classes? In particular, for the "table" scene shown in Fig.4, what is the performance reported in Table 1 when during training of the NeRFs (for both this method and LERF) only the "baseball" and "dinosaur" classes are used, and during inference it is evaluated on all 6 classes? This experiment would convincingly demonstrate that the proposed method is indeed an open-set method.
Continued discussion
Hi, reviewer kCD4 makes a good point. I understand that the authors claim that LERF and their method can be seen as taking the same inputs (images + CLIP + queries) and produce the same output (3D segments). Conceptually, the equivalence is valid in this narrow sense, but kCD4 is correct that the two methods are not directly comparable in other important senses. Specifically, LERF does allow to query any object without the need for running optimisation again, which *is* an advantage that this paper does not enjoy. I do not think the this is a huge problem for this paper. LERF is concurrent work, and the comparison is valid in the narrow sense given above. The authors should, however, make this fact *very clear* in the final version, explicitly narrowing their claim in the comparison, including clearly communicating the advantages of LERF. I.e., do not just say "we outperform LERF in this metric", but also say something on the line of "but note that our method requires solving an optimisation problem for every new set of queries, whereas LERF can respond to new queries instantly".
Thanks for your comment **L8we**. Only to clarify, my main concern is not the comparison with LERF (which is concurrent work). My main argument is the fact that the approach is advertised as open-set / open-vocabulary (which it is not, as it depends on labels during training). This may sound philosophical but has important consequences: It gives an unfair advantage to the proposed method over the baselines (since the classes are known) which makes me question the meaningfulness of the experiments. I understand that this may sound like a subtle difference (using the object classes during optimization vs. using them only later during querying/inference) however, as mentioned already, I believe it is somewhat similar to mixing training and test data in supervised methods. I don't think this paper would set a good example for potential follow up works in the emerging field of open-set 3D scene understanding. Nevertheless, I see merits in the method. I can imagine closed-set scenarios where the object classes are known in advance but no annotated training data is available -- however, in that particular scenario, the baselines have to be adapted accordingly for a fair comparison as well as the name/title of the method should be changed since this scenario does not correspond to open-set segmentation.
We thank Reviewer L8we for sharing your view on the comparison with concurrent LERF. The following response to Reviewer kCD4 is below his review section.
Thanks!
We thank the reviewer for recognition of our paper. To make the notion more clear, we will update Eq. (6) to $ R_{I_{hw}} = S_{I_{hw}} \cos \langle F_{I_{hw}}\cdot T \rangle, $ where $h,w$ denotes the index in the $H$ and $W$ channel.
The response clearly solves my concerns. Thus, I improve my final rating from 5 to 6.
Thanks!
We thank the reviewer for appreciating our paper!
Thanks for the rebuttal!
Thanks a lot for the additional explanations and experiments that clarified some of my concerns. Indeed, your answer confirmed my concern as described in weakness 1: I believe we cannot say that relying on a known set of classes is a minor limitation. This is a fundamental flaw in a method that is titled "3D Open-Vocabulary Segmentation." If the object classes need to be known in advance, can it really be called open-vocabulary? I believe, if the object-classes need to be known during training/optimization of the NeRF representation then it is the same setting as closed-world 3D segmentation where all object-classes are known during training. I think this point is currently not clearly described in the paper and a bit hidden - what do the other reviewers think about this aspect? In fact, this strong dependence on knowing the object-classes also gives an unfair advantage to the proposed method over e.g. LERF in the comparison in Table 1 (as well as all qualitative results in the supplementary). I think this comparison is unfair since LERF does not use the known object-classes. Do I see this correct? I think this aspect needs to be discussed in more detail since not all reviewers seem to be aware of the limitation. I am therefore reducing my rating until this is discussed.
Thanks for your reply. Here we address your concern in the following three aspects: 1. **Our method significantly differs from close-vocabulary segmentation methods.** Close-vocabulary segmentation methods can only segment labels from a pre-defined set of classes in a dataset. When handling new labels, they require well-annotated training data at image, region, or pixel-level depending on the new labels [1]. Our method requires only the new labels, and it does not require any annotations of the training data (similar to applying CLIP under the zero-shot setting). It’s worth noting that, unlike 2D visual recognition tasks, there is no clear definition of open vocabulary in 3D counterpart due to the lack of large-scale 3D foundation models. The typical approach distills knowledge from 2D foundation models towards 3D tasks, in which there is no clear norm to avoid using text labels during the distillation. 2. **Comparison with LERF[2].** Under the context of 3D open-vocabulary segmentation, LERF and our method require the same inputs for generating segmentation: pre-trained CLIP, DINO, multi-view images, and user-provided text labels. The user-provided text labels are also used for the same purpose – distilling knowledge from the CLIP. They are just used at different stages – LERF uses them at the inference stage and hence it does not acquire sufficient semantic knowledge for segmenting object boundaries accurately. The comparison is thus fair though the implementations of the two methods are not completely the same. 3. **Comparison with the baselines Sem(ODISE)[3,4] and Sem(OV-Seg)[3,5].** We would highlight that our setup is identical to that of the two baselines which utilize user-provided text labels in the same manner as our method. Both baselines necessitate the target text labels to acquire 2D segmentation maps before NeRF training, and they are both considered open-vocabulary segmentation methods. [1] Panoptic Lifting for 3D Scene Understanding with Neural Fields [2] LERF: Language Embedded Radiance Fields [3] In-Place Scene Labelling and Understanding with Implicit Scene Representation. [4] Open-vocabulary semantic segmentation with mask-adapted clip [5] Open vocabulary panoptic segmentation with text-to-image diffusion models.
1. I only partially agree with this point. It is correct that closed-vocabulary methods can only segment labels from a pre-defined set of classes, but the same is true for the proposed method: The proposed losses **L_RDA and L_FDA can only be computed if the (closed) set of object classes is known** prior to optimizing the NeRF, cf. "method requires only the new labels" i.e. it *does* require training labels. As soon as a method requires a closed set of labels during training, we can no longer talk about open-set, specifically because it is no longer possible to query arbitrary objects from the trained scene representation during inference, thus **it is not an open-set method**, the trained NeRF is tailored to a closed-set of classes (and needs re-training for a different set of classes). However, I see that the method does not require annotated per-point labels, so **the method can be called weakly-supervised, or unsupervised**. 2. The claim that LERF and the proposed method require the same inputs is not accurate and potentially missleading: During training of the NeRF (generating the 3D scene representation), **the proposed method additionally requires a closed set of object classes during training**, i.e., the method only works if the object classes in the scene are known. This is not the case for LERF. The claim that the class labels *``are just used at different stages"* is exaclty the point: *it is somewhat similar to using test labels during training*. In particular it gives an unfair advantage to the proposed method which solves a much easier task of segmenting a scene into a known set of classes. However, methods like LERF are more general, they compute an open-set 3D scene representation during training, that can be queried for arbitrary objects during inference. The proposed method however would require retraining every time the queries change. 3. Comparison with ODISE and OV-Seg. *"Our setup is identical to the two baselines"* - This is incorrect: In Sec. 3.6. of ODISE, it is stated that *"the test categories may be different from the training ones"*. This is not the case for this approach since each scene is optimized during training for the test labels of that particular scene (via the L_RDA, and L_FDA). There is also no experiment that would demonstrate that the method is able to segment classes not seen during training. In particular, an important experiment is missing that would potentially highlight this point: How would the proposed method perform on an unseen set of classes? In particular, for the "table" scene shown in Fig.4, what is the performance reported in Table 1 when during training of the NeRFs (for both this method and LERF) only the "baseball" and "dinosaur" classes are used, and during inference it is evaluated on all 6 classes? This experiment would convincingly demonstrate that the proposed method is indeed an open-set method.
Thanks for your exhaustive reply and suggestions. Here we have two more clarifications: **1**.We would like to reiterate that the baselines Sem(OV-Seg)[1,2] and Sem(ODISE)[1,3] also necessitate the use of text labels prior to NeRF optimization. The two baselines utilize Semantic-NeRF[1] for distillation, where Semantic-NeRF requires "some partial or noisy semantic labels for the images, such as ground truth labels for a small fraction of the images, or noisy or coarse label maps for a higher number of images." (Paragraph 5 of Section 1 in [1]). This can also be observed in their released codes at the link: https://github.com/Harry-Zhi/semantic_nerf. Similarly, we feed Semantic-NeRF with the labels predicted by ODISE[2] or OV-Seg[3] (i.e., segmentation maps of input views), which are obtained by querying ODISE or OV-Seg with user-provided texts. Afterwards, Semantic-NeRF can be trained with the output segmentation maps. Hence, our networks are trained in the similar manner as the two baselines. Regarding the statement "the test categories may be different from the training ones" from [3], it only applies to the semantic segmentation in the 2D domain (i.e., an early step in Sem(ODISE)). To distill the 2D segmentation to 3D NeRF, [3] also needs the text labels before NeRF optimization. **2**.Regarding the naming of our method, as clarified in the previous response, there are no 3D open-vocabulary foundation models (to the best of our knowledge) so similar open-vocabulary setup and approach with 2D foundation models like CLIP is not feasible in 3D space. To achieve faithful 3D open-vocabulary segmentation with CLIP, our approach necessitates the class labels of the scene without which it’s almost impossible to distill precisely 'targeted' knowledge for the **dense prediction task** of semantic segmentation (thus LERF only achieves very coarse segmentation). Hence, our work is not perfectly aligned with the definition of open vocabulary as in 2D space, but more an early exploration (and so one possible definition) of open vocabulary in 3D space while 3D foundation models are not available at this stage and even in the near future. We do agree with you that 'weakly-supervised' also defines our method well from a different perspective. We will adjust the naming of our method by either changing 'open-vocabulary' to 'weakly-supervised' as suggested or keeping 'open-vocabulary' but clearly indicating its differences from 2D open vocabulary and why we still call it open vocabulary. At last, we thank the reviewers again for your comments and insightful suggestion. [1] In-Place Scene Labelling and Understanding with Implicit Scene Representation. [2] Open-vocabulary semantic segmentation with mask-adapted clip [3] Open vocabulary panoptic segmentation with text-to-image diffusion models. [4] Panoptic Lifting for 3D Scene Understanding with Neural Fields
The author states, "To distill the 2D segmentation to 3D NeRF, ODISE [3] also needs the text labels before NeRF optimization." In fact, ODISE ultimately predicts n masks and their corresponding mask embeddings. Could you provide more details on how to obtain a complete 2D ODISE feature for an image?
Note that Semantic-NeRF requires the final **segmentation maps** instead of **image features** to distill the segmentation for NeRF. To obtain the segmentation maps from ODISE, we utilize the `demo/demo.py` script provided in their released code (https://github.com/NVlabs/ODISE). We employ the user-provided text labels and execute `demo.run_on_image()` to extract `predictions["sem_seg"]`, which serves as the output 2D segmentation map. The size of `predictions["sem_seg"]` is `[n_class, H, W]`, representing the complete 2D segmentation map for an image.
Response to author rebuttal
Thank authors for providing more visual reusts and clarifications, some of my concerns are clearly solved. However, 1. as to additional experiments on indoors scenes, I feel the methods struggle to produce reasonable segmentations compared to other scenes (probably less clutterred and richer texture). 2. I have also read all reviews, I agree with some opinions from Reviewer kCD4 that : the 'open-set 3D segmentation' is somewhat over-claimed (I understand the motivation and efforts in making language-guided 3d segmentation as discussed with reviewes), probably languided-guided/weakly-supervised, or other terms could better fit the context. Strictly speaking, a semantic prior need to be encoded in the text prompt for segmentation, compared to general promots 'a photo of something' in the classification set-up. Without such guidance, segmentation is challenging but this is partly why reviewers kCD4 insists on saying the title is over-claimed. Therefore, I would strongly encourage the authors to re-consider the title and be more conservative in using 'open-set 3d segmentation'. I do think there are still values in the method by leveraging two typical unsupervise representation models to solve the segmentation tasks, but the limited performance on indoor scenes, the importance of semantics-guidance from text prompts, as well as more rigorous re-consideration in the overall story of the paper (e.g., titles, targeted tasks) should be clearly discussed and modified within the paper. Overall, I keep my rating as 5 considering the potential value of this work but may not increase to 6 considering above limitations and unsolved issues.
Lack of comprehensive quantitative evaluation is a big concern. Although the authors added more evaluations, they are all qualitative. It still cannot improve my level of confidence about the performance of the proposed method. So, I will keep my original rating.
From the qualitative results, our method outperforms LERF by a large margin. It recognizes more objects and segments more precise boundaries, which demonstrates the superior performance of our method. We would also like to remind the reviewer that the LERF dataset lacks segmentation annotations, and the indoor datasets "suffer from defective annotations" and "some objects are difficult to predict their labels due to label ambiguity." This issue necessitates the "fixing of the label set of the dataset" for proper evaluation (as mentioned in Appendix C of FFD[1]). Consequently, quantitative evaluations on these datasets require exhaustive reannotations. We will add the quantitative results once we have completed the reannotations. [1] Decomposing NeRF for Editing via Feature Field Distillation
Decision
Accept (poster)