Volume Feature Rendering for Fast Neural Radiance Field Reconstruction

Summary

This work improves grid-based NeRF on both quality and training speed. To predict view-dependent color, they proposes to condition MLP on the volume rendered voxel features instead of each original point features. As a result, the MLP only run once for each pixel instead of the original dozen of MLP evaluations. To improve robustness, they find using the old pipeline (called pilot network) to warmup the scene for few hundreds iteration is enough. A new spherical-harmonic-based view-direction feature encoding is proposed, which combined with larger MLP achieve state-of-the-art quality. Experiments results show the superior quality and training speed from previous arts.

Strengths

The improvement is really solid and promising. The proposed method is not hard to adapt to other method. The paper is easy to follow as well. I believe this work can benefit many future researches.

Weaknesses

It's a pity that videos are not provided to showcase the improvement on view-dependent effect.

Questions

1. **Eq.4.** Is the final view-direction feature summed toghether ($\sum_{l,m} e_l^m$) like what we conventionally do for rgb or concatenated ($\mathrm{concant}(\{e_l^m | \forall l,m\})$)? 2. **Table 3.** It's better to put a table footnote that some baselines' training times are rescaled by 8 (the number of GPUs they used). As discussed in L216-219, this is not rigorous so I think it's better to inform readers who only skims the table. 3. **Table 4's H row.** It seems that the 56 should have be 256.

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

I think the limitation is well discussed.

Summary

The paper proposes a method for fast NeRF reconstruction. The main contribution of the paper is that instead of integrating radiance along the camera ray which requires evaluation of the color MLP at each point, the paper proposes to integrate features along the ray and only evaluate a single MLP on the integrated features to get the final pixel color. In this way, the proposed method can reduce the number of MLP evaluations, thus allowing the method to use achieve faster reconstruction even with a larger MLP. The larger MLP also allows the method to achieve higher-quality reconstructions. The paper does experiments on NeRF synthetic dataset and real-world 360 dataset and demonstrates higher accuracy and fast reconstruction speed than baseline methods such as instant-nap, nerfacc, and tensorf.

Strengths

1. The paper shows that integrating the features along the ray and decoding them to the pixel color with a large MLP can achieve faster and higher-quality reconstructions. 2. The paper proposes to use a small pilot network at the beginning of the optimization that applies standard volume rendering, which helps stabilize the training of the proposed method. 3. The paper did a thorough evaluation of the proposed method against different baseline methods on both synthetic and real-world datasets.

Weaknesses

See Questions.

Questions

1. It's not clear to me how the method integrates the features along the ray. In Equation 3, the paper says it performs a weighted combination of the features. I would assume that the weight is calculated in a similar way to the standard volume rendering based on the density field. Then it's not clear to me how the density field is predicted. The paper is based on the NGP representation, and it in fact uses a MLP to predict the density of each point. Is the paper doing in the same way? If that's the case, I think the paper should make it clear. Also in this case, the paper should not claim that it is performing a single evaluation NN for each pixel (or make it more clear, it's a single evaluation of color MLP). Similarly, in the case of real-world scenes, the paper uses proposal networks for sampling, which also requires additional MLP evaluations. Overall, I think the paper should make it more adequate when it says it's performing a single neural network evaluation. 2. In Line 70, the paper says SNeRG still requires many times of MLP evaluations to get the diffuse color. This is not true. SNeRG stores the diffuse colors at each voxel, and directly applies alpha compositing to get the diffuse color for the ray. Therefore, no MLP evaluation is needed. 3. The motivation behind the pilot network is kind of ad-hoc to me. First, the paper says VFR works well for most scenes but fails to converge for some scenes. If it's a scene-specific problem, It's good to know the characteristics of the scenes where it will fail. In addition, the paper says the reason for VFR not converging is that it does not have a coarse geometry to focus on samples on the surface. The first question is that why VFR needs to focus on samples on the surface. Does it mean that VFR will not work well on objects that don't have an opaque surface (like furry objects)? Why standard volume rendering does not this problem? Moreover, the paper also says larger networks tend to have more convergence issues than smaller networks, how does this relate to the lack of coarse geometry? I think the paper should have a better explanation on the convergence issues with VFR and make a more convincing argument behind the motivation of the pilot network. 4. In Figure 4, the results on the Ficus scene of Nerfacc is suspicious to me. What's the reason for the weird specularity on the pot? 5. I do have concerns over the technical contributions of the paper. As the paper said in Line 68, SNeRG has been using feature integrations for faster NeRF rendering. In addition, previous works such as StyleNeRF, EG3D, and StyleSDF have also been using feature integration for NeRF. While they don't do a systematic evaluation on the per-scene reconstruction task, I think generally feature integration has been a standard technique in the NeRF community, and has been explored in similar tasks as mentioned above. To summarize, while I appreciate the thorough experiments and evaluations of the paper, my major concern over the paper is that the proposed method has been explored in similar tasks in previous works and there is a lack of technical contributions from the paper.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

The limitations look good to me.

Summary

The authors propose to perform feature accumulation first at each pixel location, and then pass the accumulated feature to a MLP to get the rendered color in NeRF. They show improved rendering quality over baseline methods on NeRF synthetic dataset and object-centric 360-degree captures.

Strengths

1. Idea seems simple and easy to reproduce. 2. Writing is good and paper is easy to follow overall. 3. Evaluation seems extensive, though some important aspects might be missing (see bullet point 1, 3 in the weakness section).

Weaknesses

1. Will jittering happen when rendering videos using the proposed method? I feel the proposed method might be more likely to suffer from jittering issues than standard volume rendering, as the large MLP network might introduce view-consistencies. It would be great to include some video results in the demo. 2. Training time improvement doesn't seem to be that big, compared with NeRFACC that the proposed method is built upon. Is it due that volume-rendering of a relatively high-dimensional features is slower than volume-rendering of RGB colors, hence cutting down the performance gain from reducing MLP evaluations? 3. Table 2 and Table 3 only contains PSNR comparisons against baselines. I don't think this is indicative enough of image sharpness; I would like to see SSIM and LPIPS comparisons to make a better judgement.

Questions

See bullet point 1, 2, 3 in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The limitations seem to be discussed in detail in the paper.

Summary

In the paper, a novel method for view synthesis is proposed. More specifically, for given posed RGB input images, multi-resolution hash grid features are trained which are rendered to the image plane via volume rendering. The rendered feature is that passed through an MLP, processed with Spherical Harmonics (SH) feature encoding schemes, and then the final color is predicted. The main difference of the proposed system to previous NeRF models is that in 3D space, only feature grids without any MLP layers are optimized, and the view-dependent color prediction is only performed on the rendered features, i.e. "deferred shading" is applied.

Strengths

- Results: I believe the main strengths of the paper is that the proposed system leads to good results, i.e. the proposed system outperforms SOTA methods such as Zip-NeRF or Instant-NGP on the two dataset types in both, the reported training times as well as the reported view synthesis quality. - Ablation Study: A rather extensive ablation study is performed which I believe makes the manuscript stronger. It highlights the importance of the various components, and helps the reader to understand where the speed / quality improvements are coming from. - Limitation Section: The manuscript contains a rather extensive limitation section which is beneficial as it portrays a more complete impression of the method. - Manuscript organization: The manuscript is organized coherently and the information flow is rather easy to understand.

Weaknesses

- Time comparison (L. 216, L. 229): The time comparison is, if I understood this correctly, not reported with running the methods on the same hardware, e.g. different GPUs (RTX 3090 vs V100) are used, and also times are simply adopted from other papers. As the time complexity is the main selling point of this work, I believe this should be reported more accurately by running the methods on the exact same hardware / setup. - Qualitative Results - Animations: While the reported method achieves high-quality view synthesis performance measured by per-image metrics, the "smoothness" / "3d consistency" between frames is not shown. As the proposed system uses features instead of directly color predictions, the consistency between frames might be lower than for previous methods while not having a significant influence on the per-image metrics. It would there be very helpful to also show animations, e.g. in the form of fly-through videos, to compare methods with this focus. - Unclear how the “pilot network” is used: Is simply the MLP size changed, or is this operating as the “default NeRF” models, i.e. per-point MLP calls are performed, and then color/density is aggregated via volume rendering? It would be helpful if the description of the pilot network would be made more precise. - Figure 3: Unclear. What exactly is happening in the individual blocks? Is a per SH feature predicted, and then they are summed together? What is the “bottleneck arrow” indicating? Why is this skip connection actually necessary? The exact math formulations is also not contained in the respective Section 3.4. It would be helpful if the description of the deferred rendering part is more precise and clear formulas are used so that the reader can better understand the exact technique. - Density prediction: How is the density obtained? Is this stored, similarly to the features, in a 3D hash grid? - Qualitative Comparison - Figure 5: I think in Figure 5, also results from Zip-NeRF should be shown. Typos / Unclear passages: - L. 5: either in occupancy grids -> either in the form of occupancy grids - L. 8: after neural network evaluation -> after neural network evaluations - L. 19: View synthesis is a task that synthesizes -> For the task of view synthesis, unobserved views are synthesized … - L. 36: near surface -> near the surface - L. 39: standard volume rendering technique -> standard volume rendering techniques - L. 58: NN runnings -> NN calls - L. 61: to enable one sample on the surface -> to enable one sample-based rendering - L. 69: modeling the specular effect -> modeling specular effects - L. 96: represent F by use of MLPs. -> represent F by using MLPs - L. 97: as N times MLP evaluations are required -> as N MLP evaluations are required - L. 97/98: Alternatively, recent research shows that -> please add respective citation - L. 100: organized in the forms of -> organized in the form of - L. 110: the many times NN running is … -> the many NN calls are … - L. 111: the importance of a large NN in realistic rendering -> the importance of a large NNs for realistic rendering - L. 112 - 114: “Considering the fact that valuable samples on the surface are close in terms of spatial position, the queried feature vectors are also very similar as they are interpolated according to their position” -> unclear meaning - L. 213: reports on the rendering -> reports the rendering - L. 223: Using learning features organized in hash grids -> Using learned features organized in hash grids - L. 249: Larger MLP (B, C) -> B does not indicate a “larger MLP” if I understood these symbols correctly. I would also suggest to structure the names and ablation study slightly different, e.g. with the typical “Baseline”, “+ GeLU”, “+larger MLP”, etc naming convention; currently, it is very cryptic and hard to understand. Missing Citations: - PlenOctrees For Real-time Rendering of Neural Radiance Fields, Yu et al., ICCV 2021 - Plenoxels Radiance Fields without Neural Networks, Yu et al., CVPR 2022

Questions

- Are the time comparisons performed with respect to different hardware, e.g. RTX 3090 vs V100 GPUs? - Could the authors expand on the "3D consistency" / "smoothness" of in-between frames when rendering a fly-through, compared to non-feature-based methods such as ZipNeRF? - How exactly is the pilot network used? Is this predicting density and color per 3D sample point, or is this a smaller network but also operating on the rendered feature? - Could the authors explain the SH-based deferred rendering technique more precise? - Is the density predicted in 3D and stored in 3D hash grids, similar to the features?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- The authors discuss limitations of their work in the main manuscript. They mainly discuss the limitation of per-scene optimization and comparably slow / not yet real-time rendering times. I believe the limitation discussion is OK like it is; if some interesting failure mode could be illustrated with a figure, this could be interesting to add to the manuscript.

Reply to the authors

I thank the authors for the rebuttal. While I agree that feature integration is not fully explored in the multi-view Nerf reconstruction task, still I want to point out that it's a technique that has been widely used in the generative NeRF tasks such as EG3D and StyleNeRF. The real contribution of the paper is to re-study it in the context of NeRF reconstruction. However, in the current format, the paper is claiming it's introducing a new method (which is not) for volume rendering without referring to previous works mentioned above. I believe that the paper should be rewritten to better discuss the relationship with previous works and state why it's non-trivial to adapt such a technique to the NeRF reconstruction task.