UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

Summary

This paper proposes UniSDF, an improved NeuS architecture capable of reconstructing photorealistic reflective surfaces and robustly working in real-world scenes. The method models reflective color and base color as separate MLPs, using learned weights to blend them and obtain the final colors. Qualitative and quantitative results on several datasets demonstrate that the method surpasses other baselines and results in fewer geometric artifacts.

Strengths

The method proposes a new network architecture to better represent the reflective radiance field, although most of the concepts are derived from previous works. There are some original contributions, such as the blending between two fields inspired by NeRFRen, which is extended to non-planar surfaces, and the coarse-to-fine training from Neuralangelo, used here to reduce the ambiguity of reflections. Therefore, while the method is novel, its significance is not substantial. Another strength is the robust and high-quality reconstruction demonstrated in the paper and supplementary material, which surpasses other baselines. This is valuable since this problem is crucial in real-world 3D reconstruction scenarios.

Weaknesses

The biggest weakness is the lack of physical interpretation. In Eq. 8, the final color is a linear combination of the camera view and reflected view radiance fields, resulting in unclear physical meaning for each color component. Although the method focuses on novel view synthesis and geometry reconstruction regardless of the underlying physical components, this lack of physical meaning leads to ambiguity about the assumptions the authors have for the reconstructed scene. For example, the model doesn't account for how reflections change with different surface roughness, raising the question of whether it will perform well on surfaces with spatially varying roughness. Additionally, the model does not explicitly trace the reflective ray but instead models the color of the reflective ray through an MLP. This raises concerns about its ability to handle near-field lighting conditions effectively. Furthermore, I am curious about the differences between Ref-NeRF's equation c = diffuse + specular × tint and Eq. 8 c=(1−w) c_cam+w c_ref. Will these two modeling approaches yield similar performance?

Questions

1. Title: It's unclear from the abstract and introduction where the "unifying" concept comes from. "Unifying scene representation" suggests combining several scene representations, but theoretically, it is still a NeuS-based representation. 2. Missing Citations: a. MS-NeRF: Multi-space Neural Radiance Field: This paper also presents a similar idea of linearly combining different color fields. b. Several Concurrent Works Regarding Reflective NeRF: i. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections ii. SpecNeRF: Gaussian Directional Encoding for Specular Reflections c. Reflection Handling in Gaussian Splatting: i. MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections ii. Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting 3. Illustration of Existing Works: In line 131, I wouldn't call Ref-NeRF[43] as using a single radiance field, as it uses two MLPs to model diffuse color and specular color separately (though the specular MLP is conditioned on the feature of the diffuse MLP) 4. Hyperparameters in Final Loss: The final loss contains several hyperparameters for controlling the strength of each term. In my experience, these weights play an important role. It would be beneficial to conduct small ablations or illustrate how these weights were decided. 5. Input Redundancy: The f_ref and f_cam functions take in both the spatial coordinate x and the bottleneck feature b from the SDF MLP. Given that b is already a spatial feature vector, what's the point of adding x as one of the inputs? This seems a bit redundant. 6. Normals and Orientation Loss: There are two normals being produced: n is the normal from the SDF field, and n′ is the MLP-predicted normal. For Eq. 11, Ref-NeRF uses n′ for orientation loss. It's unclear if this is a typo or if n is actually used for the orientation loss. 7. Volumetric Rendering SDF: In line 124, it would be better to cite which volumetric rendering SDF system is used. I assume it's NeuS.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

There are no potential negative society impact.

Summary

The paper tackles the problem of 3D reconstruction in the presence of highly reflective objects. To address the problem, they propose to learn an SDF-based neural representation. Different from prior work, they use two radiance branches in their representation, one conditioned on the camera viewing direction rotated about the normal and one conditioned on the regular viewing direction. The outputs of the two networks are combined using a learned weight to arrive at the final rendered radiance. The authors argue that while the reflected direction is better at capturing the details in reflective surfaces, the viewing direction is more robust to real data. Through extensive experiments, they show their method is competitive with prior art in both the tasks of NVS and mesh recovery (as measured by Chamfer Distance).

Strengths

I like the problem being attempted in the paper, I also like the insight that the viewing direction conditioning is more robust, while the reflected direction produces better geometry for reflective surfaces in controlled environments. The results are also impressive. Quality: The ablation studies are well/extensively done. Clarity: The paper is very well written and easy to follow. The figures are all well understandable and the work should be easily reproducible from just the description.

Weaknesses

Significance: I think the method is generally quite a straightforward extension of existing work and the paper's contributions are fairly limited. As I see it both the radiance branches have previously existed and the Ref-NeRF parametrization could be considered more “physically based”, the authors forgo a more mathematically solid setup in favour of an increase in performance. I think the paper would benefit a bit from providing more insight into the reasons for some of the phenomena i.e. why do the weights separate like that for more specular/diffuse regions even though both are conditioned on some sort of parametrization of the viewing direction etc. I am sympathetic however to the authors that some of these phenomena just occur and might be hard to reason about, but some of the simulated experiments could provide more insight into how the optimization behaves.

Questions

1) Is there a stopgrad on any components of the normal smoothness loss (predicted vs computed normal in Eq 10)? 2) I see there is an ablation on this, but why do the authors think the weight to mix the outputs of the two radiance branches is necessary, why doesn’t this factor just get automatically learned by the two networks? 3) The authors seem to have ran ENVIDR on the Ref-NeRF real dataset (as evidenced by Fig 1?), but do not include the results in Table 2? Why? 4) Have the authors tried passing in $n^\prime$ instead of $n$ to the radiance network or also calculating $w_r$ about it? If I understand correctly $n^\prime$ is supposed to be “smoother”, perhaps this would make results more robust for just RefV?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

Limitations have been adequately discussed in the supplement.

Summary

The paper proposes a method to reconstruct scenes containing both reflective surfaces and non-reflective surfaces with high fidelity. Specifically, it trains a camera view radiance field and a reflected view radiance field separately, combining them by a learnable weight. The method is evaluated on four datasets, which covers different situations including objects and 360 scenes, shiny objects, real-world captures. And the method outperforms or is on par with SOTA on all these datasets, demonstrating its effectiveness.

Strengths

1. Extensive experiments. The method was compared to several SOTA methods and outperforms or is on par with them, showing its ability to reconstruct complicated scenes with reflective surfaces on four datasets. 2. The paper is well written. And the figures clearly serve for their purposes. 3. The author did a great job in analyzing the limitations and social impact, which is highly encouraged.

Weaknesses

1. The method involves two radiance fields and one weight field, and it has to perform volume rendering for colors from both radiance fields and the weights, which is very time-consuming even with implementation of iNGP. 2. I wonder if you have a possible explanation for why some reflected view radiance field methods is not robust to real-world scene. 3. I noticed the comparison with Factored-NeuS is only performed on DTU. Since Factored-NeuS is designed to also handle glossy objects, I wonder if how it look like when compared to this paper on Shiny Blender and Ref-NeRF real datasets. 4. As the authors mention in the section of limitations, this method requires posed images and SfM can fail on reflective surfaces. While this is a common issue, I still wonder how robust this method is to inaccurate camera pose. Even a failure case is fine.

Questions

Please see weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors listed a quite complete list of limitations. I think some are common issues for many works like requiring camera pose and dense views. With that being said, in the future the authors can consider extending this work to sparse views and how to eliminate the requirement for several fields and volume rendering, making the algorithm more efficient.

Summary

This paper proposes a new strategy for modelling view-dependent effects in Neural Radiance Field-based scene models. Existing approaches have used networks conditioned on camera view directions, as well as on reflected view directions using surface normals, but this work proposes and validates the idea that both model types should be used simultaneously. This is achieved by employing a learned weight model that mixes the two color predictions, thereby allowing regions where the color is not well explained by the reflection-based model to be handled by the view direction-based model. This is shown through ablations to be a better strategy than either strategy individually, and is also compared to a number of baseline methods which employ reflection-based modelling of view-dependent effects.

Strengths

This paper presents a simple, yet effective idea that would be easy to incorporate into other works which use reflection-based models. I think a lot of the contribution is in showing such a simple but surprising result which is not obvious to try. The quality and clarity of the writeup is high, and the evaluation seems quite thorough in comparing to other relevant works and on relevant datasets. I think the value of the proposed strategy is shown quite clearly by the experiments.

Weaknesses

The only notable weakness is that the proposed strategy is a fairly minor deviation from previously proposed methods. However, I think this is largely mitigated by the non-obvious nature of the change and the in-depth experimental evaluation.

Questions

Given the increasing prevalence of methods like 3DGS which do not use neural networks for modelling view-dependence but rather simpler approaches like spherical harmonics, do you see a way of applying the ideas of this paper to such methods?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The only limitation I see is the reduced interpretability/editability of the model compared to other approaches, but this is addressed by the authors. I see no issues with societal impact beyond generic concerns that are relevant to all scene modelling methods.