Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method

Summary

This paper use closed-form spherical harmonics (SH) color-field to regularize the training of NeRF-based models. The SH coefficient of each 3D point has a closed-form solution given a set of observed viewing angles. This work further takes the transmittance from the density field to weigh each observed angle. A residual estimation is proposed to reduce the bias due to non-uniform view angles sampling from the training views. The close-form color-field is only used to regularize the optimization of density field. Experiments show that the proposed regularization can improve quality and alleviate shape-raidance ambiguity.

Strengths

Introducing the closed-form SH coefficent solver into NeRF's optimization is an interesting aspect to improve quality of NeRF-based models. I'm interested to see if it is possible to have a closed-form optimal-PSNR color-field in the future given a fixed geometry.

Weaknesses

1. The manuscript incorrectly use the term "volumetric-based" to differentiate voxel-based methods from MLP-based methods (L76). Both of them are volumetric representations mapping spatial coordinates to modality of interest. Please fix it using "implicit" and "explicit" representations instead. 2. Some baseline results are much worse than their official report. The reported PSNR of Plenoxel on NeRF synthetic dataset is 29.83 which is much lower than their official reported 31.71 in Plenoxel's Table2. The baseline DVGO on Mic scene in Fig8 is much worse than their official results where DVGO doesn't produce the floater artifact around the mic wire. I guess the reason is the background color (from solid white to solid black) which make the shape-radiance ambiguity more severe on some scenes. It is neccessary to have a discussion about the discrepancy of the implementations and the results comparing to the original baselines. 3. The proposed closed-form color-field solution is point-wise without considering the colors from the other points on a ray. As a result, the solved color-field may not be the optimal regarding the photometric loss. This may be one of the reason why Plenoxel (which uses SH as well) have a worse PSNR when using the proposed closed-form solution. 4. Lack of theoratical backup about the bias reduction using residual. The eliminated components (supp's Eq.6) itself is estimated (supp's Eq.10) and is biased due to the non-uniform sampling. It is not clear to me how the overall bias is reduced by using the residual. 5. The ablation study only use a single scene (Fig.4). More examples can make it more convincing. 6. The quantitative improvement is incremental while it needs much more training time (1 hour vs. 10 minutes) and specialized CUDA implementation.

Questions

1. I still don't quite convinced why Plenoxel's color field get much worse PSNR when using the proposed closed-form solver comparing to the SGD solver (Fig5,L235-240). As the original Plenoxel use SH as well, I expect the closed-form solver to have a even better or similar MSE (PSNR) under the same fixed density field. Are there any other factors like the degree of SH or the grid resolution affecting the results? More discussions are necessary. 2. Missing experiments for the number of SH degree. The only discussion is Fig9 showing failure cases on high-frequency view-dependent effect. What is the quality improvement and computation cost of using more or less SH degree? 3. This work assumes the view sampling is uniform so the Monte Carlo integrator (Eq.5) has large error when the training viewing angles is not uniform (Fig.3). However, we actually know the camera poses. Is it possible to derive a pdf over viewing angles using the camera poses so that we can have a correct Monte Carlo integrator like: $$\frac{1}{K} \sum_{k=1}^K \frac{f(x_k)}{\mathrm{pdf}(x_k)} ~.$$

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The weakness doesn't discussed the discrepency between the point-wise assumption and the volume rendering (see weakness 3). If the authors decide to give up theoratical proof for the proposed bias reduction strategy, it should also be listed as one of the limitation.

Summary

While NeRFs achieve photorealistic results, they suffer from the shape and radiance ambiguity that is inherent in joint reconstruction. To alleviate this issue, the authors aim to separate the estimations using a closed-form radiance estimation. They leverage Spherical Harmonics to represent the radiance given a density field. With explicit occlusion handling based on the density field, all cameras with visibility for the point can be detected. The colors are then used to fit the SH coefficients for the specific surface point. The authors propose to use this closed-form color to regularize the NeRF training by enforcing similar behavior between the NeRF output and the closed-form color besides the regular photometric loss. The effectiveness of this formulation is shown in various experiments and is agnostic to the underlying NeRF method used.

Strengths

- One of the most significant advantages of this method is that the regularization is universally applicable to any NeRF method. - The method introduces a nice way to add a soft global view-dependent color regularization. - Besides issues with highly reflective surfaces, it improves considerably.

Weaknesses

- Currently, I have the feeling the paper would benefit from an overview figure. This can also be beneficial as a reference in the experiments, where some evaluations only rely on the closed-form color, but in others, it's used as a regularization. Here, one can create a direct link to the overview figure and the symbols for each color output. A simple overview of how the method slots in an existing NeRF framework would be a great addition and can be easily incorporated next to Fig. 1. - The authors compared against other ray distribution priors but did not compare with model priors such as RegNeRFs depth/normal input from omnidata. - Similarly, RefNeRF also aims to improve surface reconstruction. How does this method fare against the proposed one? - The authors should briefly discuss Riegler et al. - Stable View Synthesis, as it also handles aggregation of features from multiple views. The approach is vastly different, but the rough conceptual idea remains. The authors mentioned that the closed-form color field has issues approximating highly reflective objects, but the limitations are also quite severe. The performance of anything which drastically derivates from Lambertian reflection will suffer. Biasing everything towards Lambertian also solves the shape/radiance ambiguity. So I would like to see if experiments with only lower frequency view encodings or a RefNeRF with an l2 penalty on the estimated roughness are used. Minor: - In 81/82: NeRF’s representation _of_ a scene - In 82: remove ‘makes it’: The NeRF’s representation of a scene inherently suffers from shape-radiance ambiguity - In 83: error _in_ geometry. - In 86: needs to be solved - In 174, the authors used minus as a verb. The correct verb is: subtract

Questions

- What is the additional overhead of the closed-form color calculation? - Would the decrease in performance on highly reflective surfaces be mitigated with more SH bands, or would the method become instable during training?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the last point in the weakness section.

Summary

While nerfs have shown impressive properties in the ability to reconstruct the geometry of three dimensional scenes, there are still some cases where the performances are not as good as expected. This paper focuses on one of these problems, the untangling of the geometry from the color. For that, the authors propose to learn a second color model, using a close form formula based on a spherical harmonics model. On top of the classic ray tracing of nerf, this method requires to trace additional rays that goes from the query point to the cameras. The authors also show the existence of a bias depending on the distribution of view angle and propose a method to correct it.

Strengths

The paper focus on an important problem. Estimating accurately the geometry in complex scene, especially with high specularity, is still a very relevant today. Despite the heavy computational load, the authors were able to successfully train different models. They also provide results on three different datasets, with two of them being real images.

Weaknesses

The premise of the method is that density and colors are estimated independently in a nerf model. This is actually false. In particular, in the vanilla nerf model the prediction of the color directly depends on the density. The color MLP takes as input not the coordinates of the point but the features corresponding to the density predicting network. This means that nerf should actually be modeled by (c) in Figure 1 and not (a) as claimed in the paper. Many variables are not introduced properly making the reading more difficult that it should. The method is highly inefficient. For a given ray, for a given sampled point, the method requires to integrate the density along the ray toward each source camera. This means that if the model is trained using $N$ cameras and $M$ points are sampled each time to approximate the integrals, the proposed method requires to estimate the density of $NM^2$ points instead of the usual $M$ points. The parameters of the method vary between datasets. No explanation or intuition is provided for that. Regarding the experiments, the results are quite incremental with an increase of the order of 0.1dB compared to two previous methods. It's also disappointing that the analysis was not performed with more models. We can especially mention tensoRF that also has a model based on SH. It would have also been interesting to compare to methods like regnerf that adds additional regularizers to nerf to be able reconstruct better the geometry even in difficult conditions. Moreover, while the paper focuses on improving the geometry of the trained model, there is no quantitative experiment showing that the proposed method has indeed an impact on the geometry thus learned. Only a few qualitative results are presented with a color map for the depth that is very difficult to interpret. One can also regret the lack of computational analysis. No study are done on the unbiaising process proposed in the paper.

Questions

What strategy of point sampling is performed for estimating the color and transmittance of a given point (i.e when the ray from this point to the camera is drawn)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

I don't foresee any potential negative societal impact of the work.

Summary

The paper proposes a novel regularization technique for optimizing radiance fields to achieve more accurate geometry alongside realistic novel views. The regularizer is based on a closed-form estimation of spherical harmonic coefficients for view-dependent color, as a function of the training pixel colors and the predicted density field. Using this closed-form procedure to estimate color from density enables regularizing density directly through photometric loss. This regularizer is therefore adaptive to each scene, unlike more general, standard scene-agnostic regularizers like sparsity and total variation. The paper shows empirical improvement in reconstructed geometry via improved rendered depth maps, for two voxel-based scene representations (DVGO and Plenoxels).

Strengths

I really like the observation the paper makes that it is possible to predict color in closed form as a function of density field and training ray colors. It makes a lot of sense that using this closed-form solution for color and then optimizing density directly through photometric loss, should yield more accurate reconstructed density fields. And indeed the results do show improved depth renderings.

Weaknesses

presentation weaknesses - Figure 2 should include o_k and P_k, and ideally also F_c. In its current form there is a bit of a struggle to match variables in the text to components in the figure. - I don’t see much value added by Figure 3. It is used to illustrate the idea that nonuniform or systematically biased sampling can result in a systematic error in the estimated integral. In my view, this idea is clear from textual description, and the space currently dedicated to the figure could be better spent on detail (either textual or ideally graphical) about how the proposed bias correction works. - As an added point of confusion around Figure 3, why include a term that is cos(0x) (in both the figure and the accompanying text)? It appears that this is literally referring to cos(0)=1, which has no effect in the example shown. - Equation 14 (the definition of the IMRC metric) could benefit from another sentence of explanation, describing the intuition that this metric is an approximation of PSNR in 3D. - Figures 4 and 5 would benefit from more substantial captions that explain the experiment being presented and how the results should be interpreted. Currently this information is only available in the text, but it’s best if figures are also self-contained. - It’s not clear from the paper whether the results (or which subset of the results) are fine-tuned using pre-trained models vs trained from scratch. I’m not sure if the argument is that fine-tuning with the closed-form loss improves existing pre-trained geometry, or that training from scratch with the closed-form loss as a regularizer produces better geometry, or both. - There are scattered typos and minor grammatical issues; please copy-edit the final paper. evaluation weaknesses - The evaluation uses a black background, whereas in the baseline papers a white background was used. Is there a reason for changing the background color? - PSNR values reported for the two baselines (Plenoxels and DVGO) are substantially lower than the values reported in the original papers (29.83 vs 31.71 for Plenoxels, and 31.58 vs 31.95 and 32.8 for DVGO). As far as I’m aware the only difference is the use of the black background; I’m concerned that either this or some other implementation difference makes the baselines used unnecessarily weak. - I understand the IMRC metric and see why it’s useful for datasets where ground truth geometry is unavailable, but in my view an even stronger evaluation would be to compare PSNR on depth maps for a dataset where ground truth geometry is available (e.g. this could be rendered for the NeRF-synthetic dataset). - Figure 9 shows that there is a gap in quality between renderings produced by the trained color field vs the closed-form color field—can the authors comment on the origin of this gap, and how it might be closed?

Questions

My biggest question about this paper is why the proposed method is used as a regularization scheme on top of the usual least squares objective, rather than optimizing a density-only model and directly using the colors from the closed-form predictions. I wouldn’t be surprised if doing so yields even better geometry, and reduces model size as an added bonus. Or if this simpler strategy works worse than what is proposed, do the authors have an explanation why (perhaps related to the limitation illustrated in Figure 9)? I would also appreciate responses to my comments in the weaknesses section, primarily regarding evaluation weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes, limitations are discussed to some extent, but the paper would benefit from more discussion of the origin (and potential remedies, if they exist) of the limited quality of the estimated color field.