SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation

Summary

This paper tackles the problem of inverse rendering, which reconstructs geometry, material, and environmental lighting from a set of posed images with fixed lighting. It proposes two contributions: 1) representing pre-integrated illumination as a single MLP, 2) approximating self-occlusion on pre-integrated lighting and use it to supervise an occlusion MLP to disentangle shadows and materials. The proposed method achieves state-of-the-art results while being very fast to train (less than one hour).

Strengths

1. The idea of representing pre-integrated illumination as MLP is interesting. The regularization is particularly novel and clever. I really like the way how pure specular MLP (roughness=0) is used to regularize the training of the illumination MLP (Eq. 5) 2. The proposed occlusion factor estimation method is also very interesting and novel. Instead of ambient occlusion, the paper uses simple trick to factor out occlusion as an independent scalar for light integral. This scalar can be computed with MC integration and is distilled into a neural network for efficient inference. 3. The quantitative result shows that the improvement over SOTA is significant, especially on relatively diffuse scene (NeRFactor dataset).

Weaknesses

1. More datasets could be tested, e.g. the TensorIR dataset.

Questions

1. The paper mentions that the relighting results were obtained via Blender's PBR shader. Does the shader use global illumination? If so it would be unfair for some of the baselines (TensorIR, NMF) as they can only render direct illumination. A more fair way would be run another experiment where all methods do not use global illumination.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations and potential negative societal impact are discussed properly.

Summary

This paper introduces a method for digitizing real-world objects by estimating their geometry, material properties, and environmental lighting from posed images under fixed lighting. Integrating image-based lighting techniques into Neural Radiance Field (NeRF) pipelines, the method uses a scene-specific MLP for pre-integrated lighting at arbitrary resolutions. A Monte Carlo sampling-based regularizer ensures accurate lighting representation and self-occlusion predictions.

Strengths

The method proposes a method to approximate the effect of self-occlusion to improve the material estimation

Weaknesses

- The proposed method, particularly in terms of material and lighting representation, is trivial. - The correctness of the decomposed material is flawed in the qualitative results. The albedo clearly has specular and shadow bake-in, and the metalness and roughness do not match the material in the ground truth. - The occlusion introduced in the paper to improve material estimation is not robust enough to validate its contribution.

Questions

In the discussion of occlusion loss, why is only the albedo prediction addressed? The paper claims that the occlusion is designed to improve the overall material estimation.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

The proposed method decomposes lighting and material using surface reflection. However, real-world objects often have multiple material layers, such as the car showcased in the qualitative results. The clear coat reflects stronger specular light, while the lower paint layer includes both specular and diffuse reflections. The proposed model does not account for these complexities.

Summary

The paper introduces SplitNeRF, a method that integrates the split sum approximation into Neural Radiance Field (NeRF) pipelines. This approach optimizes object geometry, material properties, and environmental lighting. The method employs a Multi-Layer Perceptron (MLP) to model scene-specific pre-integrated image-based lighting at arbitrary resolutions. The model is regularized using efficient Monte Carlo sampling to accurately capture pre-integrated lighting and self-occlusion effects. Experimental results indicate that SplitNeRF achieves state-of-the-art relighting quality with only about an hour of training on a single NVIDIA A100 GPU.

Strengths

1. The method is efficient and achieves high-quality relighting results after only about an hour of training on a single NVIDIA A100 GPU, which is highly efficient compared to existing methods. 2. By using Monte Carlo sampling for regularization, the method accurately models pre-integrated lighting and self-occlusion, leading to high-fidelity reconstruction of scene geometry and material properties. The integration of split sum approximation into NeRF pipelines is novel and effective, allowing the method to disentangle environmental lighting from material properties. 3. The method demonstrates competitive performance on both synthetic and real datasets, showing its state-of-the-art performance in material decomposition and relighting.

Weaknesses

I do not have a major concern about this paper- the technical claims are sound and the paper is overall well presented. Several minor points: 1. Adding material editing visualizations in the paper could further strengthen the results and inspire downstream applications. 2. Normal could contain a significant error in e.g. lego and coffee scenes. Can you provide more insight and analysis into this?

Questions

See weaknesses 1, 2.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations discussed in the conclusions section.