GNeSF: Generalizable Neural Semantic Fields

Summary

The paper combines "generalisable" neural radiance fields and semantic radiance fields. It does so in a relatively straightforward manner, borrowing and combining ideas in these two areas, and contributing a few technical advances on top. It demonstrates good segmentation accuracy both for producing the 2D segmentation of new views and to extract 3D segmentations of the underlying scene geometry.

Strengths

The paper combines the benefits of "generalisable neural fields" and semantic neural fields, essentially inheriting the advantages of both. Compared to prior generalisable neural fields, it adds a semantic dimension, which allows to fuse 2D segmentations into the 3D reconstruction. Compared to semantic neural fields that use test-time optimisation, it can in principle be faster and use a smaller number of images.

Weaknesses

* Some key aspects of the empirical validation seem to be left untested. The main point of a "generalisable" radiance field compared to a radiance field optimised at test time is to be either (1) faster and/or (2) less data hungry. However, I do not believe the experiments test any of this. Table 1 shows that the proposed "generalisable" field is comparable (but worse) than semantic NeRF (optimisation based) on the *training scenes*. In a training phase, the "generalisable" field cannot should not provide any advantage in terms of speed or data economy compared to just fitting individual scenes. The same Table 1 shows generalisation to new scenes, which would be the main point of the "generalisable" field, but this experiment: 1. does not compare the accuracy obtained on the new scenes to performing test-time optimisation on the same scenes; 2. It does not show whether there is a speed advantage compared to test-time optimisation; 3. It does not show if there is a dat economy advantage compared to test-time optimisation. In short, we only know that the generalisable field works to some extend on new scenes, but we have no idea how that compare to test-time optimisation on those scenes, which would be the obvious alternative. * In Table 1, Ours-100 performs *better* than Ours-1000 (using 10x more data) on the training scenes. This is not entirely unexpected, but it strongly suggests that the "generalisable" neural fields take a significant hit compared to performing per-scene optimisation. This makes it even more important to check this gap on the new scenes. At the very least, the authors should discuss this aspect of the table in some depth. * line 272: The method applied on new scenes is found to outperform Mask2Former. If I understand the method correctly, thought, it is based on fusing/warping/reweighing the Mask2Former predictions using Eq. (5), even at test time. It is not the model by itself that outperforms Mask2Former, but it is the combination of the model with Mask2Former which is better than Mask2Former alone. It may be a good idea to reinforce this point at line 272. * The comparison in Table 2 is, for the most part, indicative. It compares different methods which take different kinds of inputs and are supervised in very different ways. For example, it sounds impressive that this method can be competitive with ScanComplete (2018) that takes as input 3D ground truth, but then again the latter performs segmentation based on an (incomplete) LiDAR point cloud, whereas this method uses Mask2Former segmentations form 2D images of the scene -- hardly comparable. * It is unclear how the model is trained. The authors do not mention any appearance / RGB loss, so it seems that all the various networks are trained purely with an objective function involving the prediction of segmentation masks. Is this the case, or do you also employ an appearance loss as in most prior "generalisable" radiance fields? * The goal of the paper, namely combining "generalisable" and semantic neural fields, is a bit incremental. There are elements of novelty in the specific technical details of the combination, but there aren't big surprises. * Minor issues: * line154: the "variance" $\mathbf{v}$ of the feature vectors $\mathbf{F}_i$ should be a matrix, not a vector of the same dimension. Do you mean that this is the diagonal of the covariance matrix? * line 156: why do you call $\mathbf{B}$ a *cost volume*? These are primarily just features (the only thing related to the idea of cost is the variance term, the computation of which could be weakly interpreted as comparing features from different views). * line 178: $\Delta \mathbf{d}_i$ is a vector, not a distance.

Questions

* Is the "generalisable" radiance field actually faster or less data-hungry than test-time optimisation when applied to novel scenes at test time? * The "generalisable" radiance field seems to take a significant hit in accuracy when applied to new scenes at test time. How does the accuracy compared to performing test-time optimisation (i.e., semantic NerF) on those?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

* There is no section dedicated to discuss limitations (even though the latter is recommended in the guidelines). More importantly, I could not find any substantive discussion of the limitations at all. * There is no discussion of societal impact, but I don't believe this is required for this paper.

Summary

This paper introduces a generalizable 3D scene segmentation framework based on neural implicit representation. The proposed framework takes multi-view image features and semantic maps as inputs to reconstruct 3D geometric and semantic information. The generalizability of the approach enables the synthesis of semantic maps or conducting 3D semantic segmentation for novel scenes with only 2D semantic supervision. Experimental results demonstrate comparable performance with scene-specific approaches and outperform existing the methods using only 2D annotations.

Strengths

1. This method is novel, and similar approaches have not been proposed in previous papers. 2. This task is useful and also quite innovative. 3. This paper's experiments are comprehensive and thorough.

Weaknesses

1. The article lacks clear explanations for many details, such as the detailed network structure and training strategy. This leads to poor reproducibility. 2. The article claims that only 2D supervision is required, but it does not explain how the SDF (signed distance function) is learned. If following the training pipeline in NeuralRecon or Mononeuralfusion, then this claim of "only needing 2D supervision" is incorrect.

Questions

1. Can the network learn geometry information solely based on equation (6)? If so, why? Is there an intuitive way to explain this? 2. Why does Ours-1000 perform worse than Ours-100?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

None

Summary

This paper, titled GNeSF, focuses on tackling the challenge of generalizable semantic segmentation using NeRF representation for both novel segmentation synthesis and 3D segmentation. The authors introduce several key contributions in this regard. Firstly, they propose utilizing a cost volume to construct a comprehensive global feature volume. Secondly, a pre-trained 2D segmentor is employed to extract semantic information from the source views. Finally, a soft voting mechanism is introduced to predict the ultimate segmentation label for the target pixel. These approaches collectively enhance the overall effectiveness and robustness of the segmentation process.

Strengths

1. The authors tackle a crucial problem in 3D scene understanding by extending the 2D segmentation map into the 3D space, enabling segmentation for any viewpoint. 2. The framework design presented by the authors appears logical and well-founded. Leveraging the cost volume proves to be an effective approach for encoding comprehensive scene information, while the utilization of a pre-trained 2D model allows for accurate inference of per-view semantics. 3. The introduced voting scheme successfully addresses the challenge of visibility when assigning labels to new pixels. Notably, the experimental results demonstrate a noteworthy improvement compared to previous methods, further validating its efficacy.

Weaknesses

1. Lack of references: The authors utilize the cost volume as a crucial representation for encoding 3D scene information. However, they do not provide explicit details regarding the construction process, nor do they reference important relevant papers in the deep learning era [1][2]. Furthermore, the survey conducted on the "Neural Semantic Field" appears insufficient, as it fails to mention several per-scene segmentation methods based on NeRF representation [3][4][5]. 2. Missing technical details: The meaning of "each vertex" mentioned in line 151 is unclear. Did the authors explicitly construct scene volume grids by determining the near and far plane for each view? Additionally, the process of interpolating the semantic logits (line 170) is not adequately explained. Furthermore, it is unclear why the authors utilize the distance between each point x and its camera origin (line 178). Given that this is an important contribution of the proposed method, it is highly recommended to highlight the motivation for using \delta_d at the beginning of this section. 3. Experimental results: It is unclear why the novel view synthesis quality decreases from 93.3 to 87.8 in Table 1 when including more scenes for training. Could you clarify if the term "novel view" refers to evaluating the training scenes while leaving several views for evaluation? [1] Mvsnet: Depth inference for unstructured multi-view stereo [2] Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching [3] Nerf-sos: Any-view self-supervised object segmentation on complex scenes [4] Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering [5] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation

Questions

The authors primarily focus on showcasing the results of semantic segmentation in their work. However, it is equally important to assess the quality of view synthesis (color image) when rendering novel views. It would be beneficial if the authors could compare their approach with several representative generalizable NeRF methods on the Scannet dataset. I appreciate the experimental setup and would be inclined to give a higher score if the raised concerns are adequately addressed.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See the weakness and quations above.

Summary

This paper proposed GNeSF, a generalizable neural semantic fields by incorporating contextual and semantic prior information via a U-shaped CNN and a pre-trained segmentor. Combing multi-view fused image-level features as well as soft weighting using multi-view softmax probabilities, the overall pipeline could be trained on diverse scenes and then generalize to unseen scenes with promising performance.

Strengths

- This paper proposes a valid extension to enable a generalizable NeSF model by incorporating deep image features as well as semantic priors of multiple RGB images. - Promising performance on ScanNet and Replica datasets on 2D/3D shows the effectiveness of the methods. Experiments are relative adequate.

Weaknesses

- Contributions and novelties. - As far as I know, Semantic-Ray [1] from CVPR2023 has already made attempt on learning a generalizable semantic NeRF model via attention, which shares similar spirits to GNeSF. What is the main difference? My main concerns comes from the lack of necessary comparison and discussions to related baselines such as Semantic-Ray [1] and other generalizable NeRF (NeuRay, MVSNeRF) with a semantic head in order to make the distinction clear. - Similarly, the discussion of using logits vs. prob. has also been somewhat investigated in PanopticLifting [2] from CVPR2023, thus further weakens the contributions. - As to numerical experiments, several necessary information is missing in both main paper and supplement and is a bit hard for readers to really judge the strict fairness of comparison. - Supervision of using predicted multi-channel prob tensors vs. hard-labels for baselines. If I understand correctly, multi-channel softmax probabilities are predicted per view to serve for soft voting. For baseline methods, are these multi-channel prob tensors also used for training like [2] or only hard discrete labels are used for training baselines which lost categorical information across classes? - How many nearby views are used for building the cost volumes and how does the number of views affect the final performance? - I also wonder what is the specific dimension/resolution of the built cost volume? As to cover the whole scene, I assume the cost volume owns a large spatial dimension and may make the training costly? Its dimension will affect the represented level of details of deep features. - Ablations - It seems that the mentioned design choices contributes quite marginally to the final performance as shown in Table4 - In Section 3.2, authors discusses the motivation why soft voting is used instead of directly regressing semantics. It would be good if related results could be shown in ablations (like Table4). Other minor comments which need further clarify: - It would be good to show the training expenses of GNeSF. - What is exactly the adopted generalizable NeRF in the experiments (i.e., Predict Directly)? - How to evaluate 3D segmentation as there are discrepancy between reconstructed mesh and GT mesh. Is the same procedure in Atlas used? [1]Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention CVPR 2023 [2]Panoptic Lifting for 3D Scene Understanding with Neural Fields CVPR 2023

Questions

Please see the comments in the weaknesses section above. My main concern comes from missing comparison to closely related work like Semantic-Ray and lack explanations of several key components, which makes the technical contribution not clear in my point of view.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is no clear discussion of limitations and potential negative societal impact in the main paper and supplement.

Summary

This paper proposes a generalizable 3D segmentation framework. The novel components include (a) a soft voting mechanism that incorporates image features, view difference aggregate the 2D semantic information for 2D views to 3D, (b) a visibility module to avoid being affected by occluded views. With only 2D supervision, this method allows for semantic maps synthesis and 3D semantic segmentation for novel 16 scenes. The method is also strongly supported by experiments.

Strengths

1. The ability to generalize is very important in many tasks for efficiency. This paper proposes a generalizable neural semantic field, which shows significant improvements over prior work on unseen scenes. 2. Compared to prior work that uses 3D supervision, this method only needs 2D semantic supervision during training and can be very helpful for 3D segmentation tasks whose annotations are very costly to obtain. 3. Despite being a bit complex, the proposed model includes a few novel and thoughtful components, which may be in helpful to inspire the community.

Weaknesses

1. Not enough intuition is given to explain the qualitative results. Fig. 3 shows that NeuralRecon predicts worse than the proposed method on bed. However, the authors didn't explain why this can happen (which design choice has potentially helped with this)? In the results shown in supplementary Fig. 2, it seems the proposed method has many dotted predictions (eg, red dots on the walls in the second row), what does this mean? Does it mean the proposed method may have some problem predicting the doors on the wall? 2. The clarity of the figures can be improved. For example, "MVS" in Fig. 1 is never explained in the text. The names of the features and neural networks in Fig. 1 sometimes are not exactly the same as the names used in the text, making it a bit hard for the readers to understand.

Questions

Tab. 1 shows that ours-1000 achieves a significant improvement over ours-100 on novel scenes. Have the authors tried on even larger set of training scenes? Could the authors provide some insights on when the marginal gain on expanding the training set will be negligible? How many scenes are Mask2Former and Predict Directly trained on?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicate some common patterns where the proposed method wouldl fail on.

Response to rebuttal

TL;DR: I am increasing my score, under the expectation that the authors in the final version will be more explicit about the trade-off between their generalisable NeRF and test-time optimisation (unless I am still misunderstanding something fundamental and this is actually unfair as the authors claim). * Rebuttal "In our generalizable NeSF setting, there is **no annotated training data** for novel scenes. We therefore do not perform per-scene optimization on novel scenes in Tab. 1 since it would be an unfair comparison. Nonetheless, we note that the performance of per-scene optimized NeSF is similar to Novel View in Tab. 1 (about 94%) when annotated data is available for training on novel scenes." What do you mean that there are no (2D) annotations at test time? Annotations in this paper are obtained automatically using Mask2Former, so, unless I am missing something fundamental, you absolutely can run Mask2Former at test time and then do test-time per-scene NeRF optimisation on top (basically, Semantic NeRF). If the above is correct, then your method becomes an alternative to test-time 2D semantic segmentation + optimisation, and you should measure both the advantages and disadvantages compared to the latter. Leaving the upper-right corner of Table 1 blank omits a key data point to assess it. Based on your response, I can surmise that switching to using their generalisable NeRF amounts to a drop of about 25-50 mIoU points (depending on the specific version Ours-100 or Ours-1000) compared to using Mask2Former + test-time NeRF optimisation. So generalisable NeRF is much more efficient (based on the other rebuttal responses, 4 min vs 10 hours, which is a lot!), but also quite a bit less accurate than test-time optimisation. * A: "Ours-1000 performs worse than Ours-100 (87.8 v.s. 93.3) on novel views, but much better than Ours-100 (87.8 v.s. 93.3) on novel scenes. This means that the model trained with more scenes Ours-1000 has better generalization capacity compared with ours-100. The reason is that it is easier for the model to overfit when training on 100 scenes compared to 1000 scenes. Consequently, Ours-100 performs better on novel view synthesis for training scenes, while performs worse on novel testing scenes. Please be noted that the ‘novel view’ refers to testing on the same scenes as training but different views, and ‘novel scene’ means that testing on different scenes from training." Yes, that was my intuition too when reading the paper. My point is that this is an indication that the model starts to underfit the training data at 100 scenes. There is still a test-time generalisation advantage in using more scenes, but the fact that the model underfits during training is another sign that there would be an accuracy gap compared to per-scene test-time optimisation. * A: "We respectfully disagree. Unlike other semantic NeRF methods that simply add a semantic head, our main contributions are the novel soft voting scheme and visibility module to learn a generalizable neural semantic field. With the proposed modules, our method obtains SOTA on both semantic view synthesis and 3D semantic segmentation. On 3D semantic segmentation, our approach with only 2D annotations can even outperform existing strong supervision-based approaches." I recognise that, while the basic problem setup and starting point in formulating a solution a fairly "natural" combination of prior works, making this work in practice is not trivial. When I said that the work feels to me a "bit incremental", I meant that there is good technical progress in the paper, but this progress is not revolutionary.

Response to rebuttal

After reading the rebuttal from authors, I decide to keep my score and lean towards positive.

Comments on Rebuttal

I appreciate the efforts of authors in clarification towards recent generalizable semantic nerf which addressed some of my concerns. I have a few questions left, 1. As to the numerical comparison to S-Ray, especially on the set-up of training on 1201 scenes and test on 312 scenes, are the set-ups of S-Ray properly adjusted in case the comparison may not be strictly fair? I mean, for example, is the number of trainging steps (and other hyper-params) properly increased since it is initially trained for 260k steps of 60 scenes. Are the same M2F labels used (the same to GNeSF) as training signals for S-ray? 2. I am wondering if the main performance inrease on many unseen scenes comes from the adoptation of using a **cost volume** to explicitly accumalte MVS information (appearance, semantics, geometry, etc) compared to method like S-ray which purely uses a MLP for representation? As a consequence, I am not sure whether the cost volume contributes more than the voting strategy itself. In addition, by using such design choice I find the training expenses greater than methods like S-ray using a single 3090 GPU, though it is not really a weakness of the proposed method.

Comment by AC

Dear authors, Thank you for your detailed responses. In my opinion, the authors' response clarifies the questions **Q1** to **Q5**. Please update the final version correspondingly to improve the clarity. Regarding **Q6**, I agree with the authors that novel view appearance synthesis is not the main focus of this paper, and this is clear from the abstract and the introduction. Hence, being comparable with most generalizable NeRF methods is fine. But I have some clarification questions regarding the results: 1) It is interesting that the proposed method has the lowest LPIPS despite not achieving the best PSNR. The SSIM is clearly worse than Sufel-NeRF while the LPIPS is better. What could be the reason? 2) It is also not clear why the proposed method outperforms IBRNet. Could the authors further elaborate on this? @Reviewer 1Moh, please let us know your thoughts if you have further comments. Best, AC