GenS: Generalizable Neural Surface Reconstruction from Multi-View Images

Summary

This paper proposed a method for neural implicit surface reconstruction from multi-view images. The proposed method is generalizable without per-scene overfitting. The key component of the proposed method is a multi-view feature volume constructed from a multi-scale FPN network. During training process, the multi-scale feature is used to constrain the view-consistency as well as geometry smoothness. Experiment results show that the proposed method achieves better results than some previous pre-scene methods and achieve state-of-the-art results on DTU dataset with finetuning.

Strengths

- A new approach that targets generalizable neural surface representation is interesting. The proposed method is reasonable and technical sound. - Writing is overall clear. - Result quality is not bad.

Weaknesses

- The current proposed method (from my POV) is a direct extension of SparseNeuS with more input view and multi-scale volumes (and two additional loss term that using multi-scale features). Neural geometry representation with multi-scale approaches has been also proposed in [1] (BTW this was also missing in related works of this paper). The combination of these methods has not appeared before, but the overall novelty would still be limited, as the paper does not demonstrate the key challenges (either theoretical or practical) when combining these techniques. It more sounds like "we put this and that together, and it works better than previous methods." - Experiment is a little bit weak. Quantitative results are only reported on DTU dataset. From results on Tab.1, retraining Geo-NeuS with the same dense-view input works really well (almost same as GeNIS with finetune). Fig.8 demonstrates qualitative results on BlendedMVS dataset; however, it is hard to say which one is better for Ours(ft) and SparseNeuS(ft). These results suggest the improvement of the proposed method (especially, over its most related baseline, SparseNeuS) is marginal. [1] Takikawa, Towaki, et al. "Neural geometric level of detail: Real-time rendering with implicit 3D shapes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Questions

- How long does it take for finetuning? A long finetuning process would make the proposed method less interesting for "generalizable" representation. - I still failed to understand why the proposed view-contrast loss can improve smoothness. Is the teacher volume and student volume constructed with the same network? And why add consistency between student and teacher can improve smoothness? - I notice that ablation studies are only conducted on three data points instead of full set in DTU dataset. Is there any reason? - Is multi-view volume a must for generalizable representations? Current experiments only demonstrate multi-view feature consistency is useful. What about using single volume but with all the feature consistency and contrast constraints? i.e., a naive extension of SparseNeuS with the loss proposed in the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations and societal impact have been discussed in the paper.

Summary

This paper proposed a generalizable Nerf method. The novelty of this method lies in a proposed multi-scale feature-metric consistency loss and a proposed view contrast loss. The feature-metric consistency loss encourages the features extracted from different images at the same surface point to be consistent with each other. Doing this in feature space is claimed to be more discriminative. The view contrastive loss enforces that the recovered surfaces from subsampled image sets be aligned with that of a full image set, to encourage smoothness n surface regions with poor visibility. This method can be applied directly to predict surfaces or have its output fine-tuned for a short time to arrive at a stronger final solution. With this method, the authors achieve state-of-the-art results on the DTU and BlendedMVS datasets.

Strengths

The strengths of this paper are that the proposed losses are novel and well-founded and that their inclusion in a generalizable Nerf pipeline leads to state-of-the-art results both qualitatively and quantitatively.

Weaknesses

The main weakness of the paper is the grammar and sentence structure. I would heavily advise that some polishing be done to the paper before the final submission. Other than that the proposed method seems sound and the results are impressive.

Questions

Some requests I have for the paper: -> Provide a numerical ablation over removing the proposed losses -> Provide standard deviations in the means for Table 1 -> Bold the best-performing model in each column for Table 1 With respect to comparing to prior works, why do you not provide the performance of models such as SparseNeus and PixelNerf in Table 1? You make claims that these will perform worse, but this is not demonstrated quantitatively. Please provide the results of training at least one other generalizable method with the same fine-tuning process applied to your method in this table.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

-

Summary

This paper aims to address the problem of multi-view reconstruction using neural implicit representation. The key to improving both efficiency and efficacy is a deep network that predicts a generalizable multi-scale feature volume encapsulating the geometry and appearance of the scene given a set of posed images. The generalizable feature volume circumvent the need of optimizing the neural imiplicit representation from the scratch for each testing scene. The authors also devise two losses -- multi-view feature consistency and view contrast loss -- to further improve the reconstruction performance. Experiments demonstrate that the proposed method outperform several state-of-the-art methods in the DTU dataset.

Strengths

- View contrast loss is novel and interesting in the context of NeRF training. - Ablation studies on multi-scale feature volume, , multi-view feature consistency, and view contrast loss validates the contribution of these critical components in the proposed system.

Weaknesses

- My major concern of this paper is that the key ideas presented have been explored separately in several works. 1. Multi-scale feature volume for neural implicit field has been widely used in MonoSDF, NGP, and several other works. Generalizable multi-scale feature volume has been a go-to representation for deep learning based Multi-view Stereo. 2. Feature consistency has been investigated in MVSDF. 3. The surface approximation method is borrowed from Geo-NeuS. As a result, the novelties are somewhat limited. - What is the annealing factor in (eq. 5) in training? I understand the authors provided a detailed description of the annealing factor for fine-tuning on a particular scene, but I wonder how to control the annealing factor in a batch-wise training setup. - Some questions regarding experiments: 1. The baseline results in Table 2 are surprisingly low. Without equipping all the novelties presented in the paper, the baseline seems to be a NeuS with a large feature volume. Can the authors explain why this baseline is much worse than NeuS (e.g., 3.7 vs 1 in scene 24 in DTU)? 2. Missing comparisons to recent efficient neural implicit methods. Although works like Voxurf do not have a generalizable deep network to obtain learnt feature volume given a new scene, they also claim to achieve state-of-the-art results on DTU after optimizing on a scene for several minutes.

Questions

See questions in the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper proposes a generalisable neural surface reconstruction approach that is able to learn an SDF function from volume features and does not need per-scene training.

Strengths

1. This paper uses a multi-scale volume to model the feature field. 2. The view contrast loss is novel to me which helps to enhance the reconstruction quality over regions that lack multi-view observations.

Weaknesses

1. This paper does not mention some of the latest advanced approaches that also work on the generalisable neural surface reconstruction: VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction Rethinking Rendering in Generalizable Neural Surface Reconstruction: A Learning-based Solution 2. The experiments are not complete. It does not compare with the two SOTA methods mentioned above. Also it does not compare with the learned MVS methods, such as MVSNet. 3. The results of the reconstruction without fine-tuning is actually not looking good enough to me. For example, in figure 8, the mesh results are quite noisy.

Questions

1. In Eq. 2, the MLP also takes position as input. Doesn't it make the function not generalisable to other scenes, since position is scene-specific? 2. In Eq. 3, does it make sense to blend the colors of all view projections? What if the point is not visible in certain views? 3. Eq. 10 is proposed to encourage multi-view feature consistency. But how will it handle occlusion cases? If there are occlusions, the features are naturally inconsistent.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are already revealed in the paper.

reply

My comments and questions have been adequately addressed. Thank you for taking the time to do so. I believe my score is still fair for the paper.

Reply to rebuttal

Thanks for the detailed rebuttal, here is my comments after reading the rebuttal. - Related work (Q1). I appreciate the rebuttal discussed the differences between this work and the mentioned work. I would like to additionally emphasize that the Neural LOD contains octree structures that are not easy to construct from the image feature (which seems to be an advantage of this proposed method). - Key challenges and method significance (Q2, Q5, Q6, Q9): I am now convinced that the proposed multi-scale view-contrastive loss is useful and interesting. It seems this strategy helps the learnt representations robust to sparse inputs. - Experiments (Q3, Q4, Q7, Q8). I appreciate clarifications in the rebuttal. Overall, I feel positive after reading the rebuttal and I am willing to increase my score to "Weak Accept". Thanks.