NeRF Revisited: Fixing Quadrature Instability in Volume Rendering

Summary

This paper describes a modification to the way NeRF performs approximate integration for volume rendering. Instead of assuming piecewise constant opacity, they propose to use piecewise linear opacity. They argue that this resolves what they dub the "quadrature instability" in NeRF. They re-derive the quadrature integration method using piecewise linear opacity and piecewise constant color, and show that this also enables them to derive a precise inverse of the ray termination CDF for importance sampling. Through experiments on standard NeRF datasets they show an quantitative improvement in rendering quality and also show qualitatively that their method makes the rendering sharper and more stable across change of viewpoint and camera distance.

Strengths

This paper addresses an often overlooked aspect of NeRF which is the specifics of how the volume rendering is implemented and whether it could be improved. The propose what to the best of my knowledge is a novel modification, which is to approximate the opacity along the ray as piecewise linear (instead of piecewise constant) in order to obtain a more accurate quadrature estimate of integral. This formulation also leads them to derive a more precise method of importance sampling. They analyze conceptually how the piecewise constant assumption in NeRF leads to conflicting ray supervision between perpendicular and grazing angle rays. They claim that their piecewise linear formulation reduces this problem and leads to a more peaked opacity PDF. They provide extensive derivations for the formulas in the supplemental material. The new formulations are a "drop-in" replacement for the equations used in vanilla NeRF, and thus they can easily modify existing NeRF implementations to use them. In their evaluation they compare against vanilla NeRF on the standard synthetic and real forward-facing datasets and show a quantitative and qualitative improvement in rendering quality across the board. They also show that their method improves SCADE, a recent method that incorporates depth estimates into NeRF training, and show that swapping in their formulas leads to a modest improvement in quality. They hypothesize that this is because of their more precise importance sampling.

Weaknesses

They seemed to have missed a highly relevant previous work: Wu, L., Lee, J. Y., Bhattad, A., Wang, Y. X., & Forsyth, D. (2022). DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16200-16209). This method is based on a voxel grid approach rather than an MLP. However, the DIVeR paper also discusses the inaccuracy of integration using piecewise constant opacity. Their solution allows for integration over more complicated functions than piecewise linear and so might be more accurate than the piecewise linear approach proposed here. Indeed, DIVeR reports higher average rendering quality metrics on the NeRF synthetic dataset than what is reported here. Because they claim that their method improves sampling and makes the opacity function more peaked at the surface, I would be very curious to see a comparison of surface reconstruction quality compared to NeuS: Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., & Wang, W. (2021). NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems, 34, 27171-27183. The explanation of inverse transform sampling in the original NeRF method (section 3.2) should be expanded, as this is not explained thoroughly in the original NeRF paper. Their discussion (L175-178) is quite brief considering how critical it is to the paper. The paper has many grammatical errors and incomplete sentences which make it difficult to read. I included some examples here: * L135 "Hence the continuous probability density function (PDF) the ray r(s)" * L138 "s is a point on the ray r" -> it doesn't make sense that s would be a point (since it is a scalar) * L152-153 "P_j is the probability of each interval, which is mathematically equivalent to the probability of the interval." This sentence sounds like a meaningless tautology. * L154 missing a period * L117 "Then taking x = g^-1(u)." This is an incomplete sentence. * L117-118 "However, this does not necessarily result in the samples from the actual ray distribution p(s) from the model." This sentence doesn't work grammatically. Some more analysis of why this doesn't result in sampling from the distribution would be helpful (explanations and visualizations). * L203 "such an example of a sample-based loss used for NeRFs is depth" * L232-3 "we pointed the drawback" * L238 "thus the resulting CDF F being continuous" * Table 1 caption: needs a period * L273 capitalize Lego

Questions

I think the most critical issue is the comparison with DIVeR. How does this method compare to DIVeR, which appears to support an even more accurate computation of the integral? I would also be interested to know whether they expect that this method will improve the surface reconstruction capability of NeRF, and how it compares to NeuS in this regard. Finally I would like to know if they have a reference regarding the original NeRF's method of inverse transform sampling the CDF -- is this just based on the NeRF code or is there a reference for that?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

A section on limitations does not appear in the paper. A frank discussion of the limitations of the approach would strengthen the paper.

Summary

Neural rendering methods like nerf relies on the integration of contributions (both color and density) along rays to predict views. Since analytical computation of the integral is not possible (the color and the density being both prediction from neural networks, usually MLPs), an approximation is estimated. Traditionally, the hypothesis done is that the color and the density are locally constant thus leading to a rectangle rule approximation. In this work, the authors propose to replace the the constant approximation of the density by a linear approximation. This allows for a more accurate estimation and a closed-form formula for sampling points during training.

Strengths

The proposed modification is simple and elegant, it is mathematically driven leading to a closed-form formula. Moreover it is plug-and-play with most (all?) popular nerf-based methods and have a direct impact on the quality of the density estimation. It offers an important solution to the fuzzyness problem of the density learned by nerf methods, thus allowing for better surface reconstruction, and especially smaller details. The paper is clear and well written. It introduces well all the different concepts necessary to understand the contribution. Experimental results are convincing with a non-negligible improvement of 0.5dB in PSNR but also across other metrics (SSIM and LPIPS) on multiple classic datasets (one synthetic and two real).

Weaknesses

While the paper is very clear and the ideas elegant, I find the experimental section lacking: * The proposed modification should impact mostly the density and thus the geometry of the reconstruction. The experimental section only focuses on a. While this is can be sufficient, I find disappointing that the authors did not take the opportunity to show the potential improvement of the geometry for the different datasets showed in the paper. * The choice of baseline nerf is surprising. Given the type of experiments (with large changes of camera-to-scene distances), I think that mip-nerf would have made for a much better baselines. It already fixes a large number of artifacts pointed out by the authors in Figure S1 and therefore would have been a more appropriate comparison. I nonetheless expect that the conclusions can be transcribed to more recent frameworks. * I find surprising that there is barely any discussion about the computation complexity of the proposed sampling (nor time measurements). I saw later that it is mentioned in a "Limitations" section in the supplementary but for me it's not a limitation per say and it should be studied in an ablation inside the paper. Especially since S1.1 shows that a better sampling can provide interesting convergence properties, thus allowing to change the number of samples used during the training. I would have expected a comparison with less samples than the 64 + 128 baseline (64 + 64 seems like an obvious thing to try based on the comments inside the text, I'm surprised that authors didn't even try that), to see the impact and find maybe a better compromise between computation time and performance. looking at the convergence plots would have also been interesting. Indeed, the proposed method might offer a faster convergence compared to the previous sampling, thus having a double impact. * While slightly outside of the scope of this paper, it still would have been interesting to show the impact on the more practical methods (instant-ngp or voxel based methods for example) that are often preferred because of their fast training speed.

Questions

RFF are more often referred to as LLFF. I suggest renaming to avoid any confusion (even though LLFF technically refers to the method proposed in the same paper) since a reader would expect that name.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Given the nature of the contributions, there's no major limitations nor potential negative societal impact to discuss. Some limitations are mentioned in the supplementary material but they are more related to the analysis of the method as I mentioned in "Weaknesses".

Summary

This paper presents a fix for the "quadrature instability" arising from sampling of points for numerical integration used by NeRF2020 and its successors.The sampling inconsistence may not be big on classical rendering techniques, but can be significant when using neural architectures. Opacity and color values are assumed to be piecewise constant by NeRF. This paper offers a tractable solution with piecewise linear opacity values and piecewise constant color values. To avoid dense sampling the NeRF and follow up use a importance sampling approach based on corse-to-fine strategy. For drawing these importance samples the NeRF utilizes a surrogate CDF as original CDF is not invertible. The impact is shown on several standard scenes on difficult situations such as close to surfaces, fine feature areas, etc.

Strengths

The proposed method is effective and efficient, drawing on existing volume-rendering literature. The modification suggested is rather simple but effective. The modified method should improve the quality of all Radiance Fields methods and hence of interest to researchers working on those areas.

Weaknesses

The paper proposes a small, clean formulation of ray-integration that improves quality of NeRF reconstruction. The limited nature of the idea is its weakness, if it can be called so. The improvements in quantitative measures like PSNR is marginal, but the qualitative improvements are significant in particularly difficult situations. The usefulness and utility of the formulation could be established more clearly by incorporating it to other NeRF variants whose code is available. I would wholeheartedly support this idea in a vision conference like ICCV/CVPR. It is upto NeurIPS to decide if such a small-but-effective idea is of sufficient interest to its audience.

Questions

Here are some questions/points: - Can this method be integrated into all the other Radiance Field methods that essentially use the same ray-tracing formulation? Has it been done? That will increase its appeal - What is the computational overhead of the expanded formulation compared to vanilla NeRF? The paper mentions 18 hours for 500K iterations. I am interested in knowing about head-to-head comparison with vanilla NeRF to know if there is significant performance penalty in using this formulation. I would like to know it for both training and rendering for view generation. - Just curious: What is the adverse impact of using piecewise constant color values? What could the advantage be if we use linear color also? Does the other combination of piecewise constant density and piecewise linear color have similar closed-form solutions? What would the visual impact of that combination? - Are comparisons with later NeRF such as ZipNeRF/MipNeRF available? It would be interesting to see if the new proposed strategy scales well to those as well. - Depth supervised experiments seem valid. Does this pan-well against works like RGBDNeRF (Yuan et al. 2022 TPAMI)? Minor points: - Citations appear in ordinary parentheses as "(25)" instead of the more standard "[25]". This was very confusing in the beginning. Is this OK with NeurIPS style? - Line 164: Isn't it more correctly R^3 x S^2 --> R^3? Why discretize it to [0-255]? Are you doing something different here?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

What are the limitations of this approach in the authors' views? None are mentioned in the paper. Showing the formulations effectiveness on other Radiance Field recovery methods (they all use the same rendering equation) will enhance the paper.

Summary

The paper addresses a fundamental limitation of existing NeRF formulation, namely piecewise constant integration, which results in sensitivity to point sampling, summarized as quadrature instability in this paper. To address this, the paper proposes a new formulation based on piecewise linear quadrature for density and piecewise constant quadrature for color. The paper demonstrates that the proposed method produces sharper and more stable results, and leads to better quantitative metrics.

Strengths

The paper clearly analyzes the problem of existing methods. The proposed solution is theoretically sound. The proposed method leads to improvements in NeRF reconstruction, both qualitatively and quantitatively. The paper shows detailed theoretical analysis and visual comparisons in supplementary material and video.

Weaknesses

The improvements appear to be quite small from the overall quantitative metrics. It seems that the improvements may be more visible in specific scenarios, but may be subtle for general inputs.

Questions

Is visual difference obvious for a random input, or if the example shown needs to be carefully chosen?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not discuss its limitations. It may happen that the method is robust, but some limitations (including limited improvements) in certain scenarios should still be discussed.