PyNeRF: Pyramidal Neural Radiance Fields

Summary

This works tackles the problem of antialiasing in grid based Neural Radiance Field representations (e.g. INGP, DirectVoxGo). To this end, a very simple solution is proposed: *instead of training a single NeRF with multiscale features, train separate NeRFs at different resolutions and decide which to use based on the distance of the sample from the camera (size of the pixel footprint)*. Specifically, two levels (closest smaller and larger) are used to predict the color and density, which are then linearly combined into a single prediction per sample along the ray. This requires one additional MLP evaluation per each sample, but is still much more efficient than the multisamples proposed e.g. in ZipNeRF. The proposed method is evaluated on the Multiscale Blender and Multiscale Mip-Nerf360 datasets, where it consistently outperforms the state of the art.

Strengths

In my opinion, the main strength of this paper is its simplicity. The proposed idea is very general and could easily be implemented in the grid based representation at only a minor cost (one additional MLP evaluation per each sample). The increased storage is also not a large problem, as shown in the supplementary, as features can be shared across level resulting only in the overhead of MLP weights (very small compared to the feature volumes). I really like such simple ideas that lead to good performance improvement. The paper is also well presented and easy to understand, the ablation studies support the main ideas (Pyramid training, simplification of the Laplacian pyramid). I especially liked the analysis in the supplementary material (it is maybe unfortunate to have that only in the supplement and could maybe be moved to the main paper for the camera ready version).

Weaknesses

I only have one minor weakness - sometimes some details are missing, or the description is slightly ambiguous. For example, the number of levels, size of the features per level, MLP dimensions are missing. In the supplementary, it is not fully clear to me what the shared INGP table denotes? In the original INGP each level has its own hash-table, does this imply that a single one is used for all level (what is the dimension and number of the features in this hash-table?)

Questions

**Comments - potential improvement**: - The proposed method is very general, but it is only evaluated on the INGP representation. While TensorRF is used in the ablation in the supplementary, it is not clear if the proposed multilevel training also improves its performance. It would be great to support the claims of the generality by applying the ideas to at least one more representation. **Questions**: - The gradients for each camera sample are only propagated back to 2 levels. If the camera poses are very different, this could mean that in some areas the finest levels are never supervised - what happens if the novel view is closer than camera of the training view? It would mean that the levels that are sampled were never supervised or? - In the representation in the main paper, I don't fully understand why the storage requirements are larger? Is it because there is no concatenation of the features, and hence the feature vectors of each level are of higher dimension? It would be good to clarify this - If I understand correctly, the method proposed in the supplementary with the shared features across the levels is even simpler (only adds L-1 MLPs to the formulation) and achieves comparable performance. Is there are good reason to not make that the main method? I guess it is not trivial to share the features in the methods that store them explicitly (tensorRF or DirectVoxGo).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors describe the limitations of their method and potential negative societal impacts adequately.

Summary

PyNeRF replaces the implicit representation in Mip-NeRF with a voxel-based representation method, which combines the cone sampling method and explicit structural representation by interpolating on different voxels based on coordinates of different scales. This method can be easily applied to existing accelerated NeRF methods. PyNeRF improves training speed by 60 times compared to Mip-NeRF while ensuring anti-aliasing effects and reducing errors by 20%.

Strengths

* Experiments are performed on multiple datasets, including synthetic, real, and large-scale datasets, which demonstrate the superior performance and training speed of the proposed method. PyNeRF combines the anti-aliasing capability of MiP-NeRF with the fast fitting of explicit structure, showcasing the advantages of the two methods. * Three strategies are adopted for the interpolation method (GaussPyNeRF, LaplacianPyNeRF, pyNeRF), demonstrating that the PyNeRF approach can save storage, improve rendering speed, and ensure the same rendering quality as interpolating on multiple voxels. This verifies the rationality of PyNeRF's interpolation strategy.

Weaknesses

* The paper states that the proposed method can be applied to existing accelerated NeRF approaches (L9). However, the results show only slight improvements when applied to other acceleration methods (PyNeRF - TensoRF-CP, PyNeRF - TensoRF-VM in Table 5 of the supplementary). It appears that the use of hash tables is the only approach that shows significant improvement. * Although the multi-scale sampling and interpolation methods in this paper play a role in anti-aliasing and accelerating training speed, they also increase the consumption of storage space. According to Table 5, using a shared hash table can achieve the same performance, so why not use a shared hash table? This would save storage space. * Moreover, adopted methods, e.g. multiscale sampling and multi-resolution voxels, have been proposed and widely used in many existing methods, which makes the novelty somewhat limited.

Questions

* Further analysis is needed to demonstrate why the gain on TensoRF is marginal. * The feasibility of the method can be tested on more accelerated NeRF approaches. The storage size of this method may be the main factor that impedes its wider application.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have addressed the limitations.

Summary

The authors introduce a pyramidal radiance field reconstruction method, which reuses multi-scale feature grid representation and area matching algorithm for level indexing. Specifically, the method trains a pyramid of models at different scales and interpolates point features between neighboring levels determined by the cone size to which the point belongs to. The effectiveness of the proposed method is evaluated on the multi-resolution version of the blender, ADOP and Argoverse sensor dataset. The experiments show that the proposed method is able to provide better rendering quality while preserving high reconstruction speed.

Strengths

1. The motivation is clear. The manuscript addresses an interesting and important anti-aliasing problem for grid-based representation. It's good to a paper on that. 2. The manuscript is well-written. Figure 1 and algorithm 1 are very helpful for understanding the manuscript, good job. 3. The quantitive results reported in the tables are quite good.

Weaknesses

1. The quantitive result on the standard resolution is missing, such as single resolution for nerf synthetic, mipnerf 360, leading it hard to judge the general performance. 2. The optimization progress is unclear to me. Is the different level feature optimized with the specific scale of the dataset (L113-115) or supervised only on the final rendering (L129)? 3. Experiment details are missing, such as level numbers used for each dataset, the grid resolutions, and the feature channel, representation (dense grid?). 4. Since the manuscript is focused on anti-aliasing, but I could not find the zoom-in results in the appendix video, are there flickers when rendering sequences?

Questions

1. Can this representation be trained and evaluated on a standard multi-view dataset? 2. Table 1 and 2, any insights on why the quantitive (Plenoxels, K-Planes, TensoRF and iNGP)) and the rendering video of iNGP is much worse than the result reported in the original papers? Is that because they are evaluated on multi-resolution? In this case, I would expect the rendering quality to be the same.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors provide a limitation section and make sense to me.

Summary

This work presents a method for anti-aliased renderings for grid-based NeRF representations by jointly optimizing a hierarchy of coarse-to-fine grids. The idea is neat and well justified by empirical evaluations that show quantitative and qualitative improvements over baselines.

Strengths

1. The proposed method seems simple and easy to reproduce. 2. Experiments are extensive, and demonstrate the effectiveness of the proposed method.

Weaknesses

1. The method is built upon the nerfstudio library. But the empirical results of that base model seems missing. Would be great to include those results to help better see the improvements made by the proposed anti-aliasing techniques.

Questions

1. It says that the method is implemented on top of the nerfstudio library. Which specific model was built upon? Is it the nerfacto one, or others? 2. What do the numbers 11.2,8.6,5.7 mean in Fig. 3(a)? 3. Is the MLP shared across the different mipmap levels? What is the MLP size? 4. Does the method apply to other grid-based representations like TensoRF, or K-planes?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are adequately addressed.

Summary

This paper presents a method to address the aliasing artifacts in grid-based NeRF. It introduces a pyramid of grids of different resolutions to represent a scene. To query the color and density of a 3D point with a certain integration volume, the method finds the two pyramid levels that best describe the point and performs a weighted combination between the outputs of the two levels. Experiments on both synthetic and real datasets demonstrate the advantages of PyNeRF over previous methods in rendering quality and speed.

Strengths

- How to address alias artifacts is an important problem in NeRF. While mip-NeRF proposes an elegant solution for positional-encoding-based NeRF, extending this to grid-based NeRF is not straightforward. This paper presents a reasonable and novel approach to address the aliasing issue in grid-based NeRF. This is achieved by maintaining a grid pyramid with different levels of resolutions and only using suitable levels during rendering. - The proposed PyNeRF shows clear advantages over previous approaches, as demonstrated in experiments. The rendering accuracy is significantly better than other grid-based NeRF and in most cases better than mip-NeRF. Qualitative results also show that PyNeRF better captures fine details. The running speed is on par with other grid-based NeRF and is much faster than MLP-based NeRF. - The effectiveness of the proposed design choice is verified in ablation studies.

Weaknesses

- The technical contribution is weak, as the proposed method is rather simple and does not involve much technical challenge. The iNGP already has a multi-resolution grid and this work simply chooses to use the two suitable for the scale of the point. Although this strategy is shown to be effective, I doubt if the technical contribution is enough to be presented at NeurIPS. - Some descriptions are not clear or reasonable. For example: - The sentence on Line 121 and 122 is unclear. It's not well explained how P(x) is calculated. It seems to be the projected area from the voxel to the 2D plane. If this is the case, then it is a somewhat odd design. A voxel is a cube, thus its projected area to the 2D plane is view-dependent. But the selection of scale level should not depend on view but depend on the distance between the 3D point and the camera. - The first version of the method is called GaussPyNeRF. However, it is not explained how the method is related to Gaussian. I do not see a clear connection. - Line 156: "We can quickly train these along with our initial low-resolution model and then use them to train higher-resolution levels in a sample-efficient manner." It's not clearly described how this is done.

Questions

- Authors can respond to my concerns mentioned in weaknesses. - Line 155: "many contemporary NeRF methods use occupancy grids or proposal networks to generate refined samples near surfaces". I suggest adding related citations here. - Line 35: fast-rendering approach -> grid-based approach. Fast rendering includes not only grid-based approaches but also others like point-based approaches. This paper targets grid-based approaches. - The title does not mention anti-aliasing, which is the main problem to be solved in this paper. A minor suggestion is to mention it in the title.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Response to the rebuttal

Thank you for clarify some details, I think that adding the discussion on the zoom in/out effect to the supplementary would help strenghten the paper. **Storage**: This is still somewhat confusing to me, but let me see if I understand correctly. Let's assume an INGP configuration with 16 levels, the original works will store 16 distinct hash tables (one per each level) with e.g. $2^{19}$ features each of dimension 2. When querying the features, features of each level will be interpolated trilenearly first and then concatenated to a 32 dim feature vector (16 levels of dim 2). If I understand correctly in your base case you have the following hash tables: for grid resolution 1 (1 hash tables with dim xx), for grid resolution 2 (2 hash tables with dim xx), ... for grid resolution 16 (16 hash tables with feature dim xx)? This would also mean that the dimension of the input to the MLP changes based on the level? What is the xx in this case? In the storage optimized version, you instead have the same as INGP, i.e. 16 distinct hash tables (one per each level) with e.g. $2^{19}$ features each of dimension 2. And you only index into a different MLP based on the grid resolution?

Response to the rebuttal

Thank you for the clarifications. The rebuttal addresses most of my concerns and I remain positive about this paper. Please make these revisions accordingly in the revised version.

final rating

Thanks for the rebuttal and clarification, I am leaning toward the original "Weak Accept" rating.

Thanks and Final Rating

Thanks for the authors' rebuttal which has addressed my concerns. I would like to raise my rating. Besides, it is highly recommended to include the additional results and contents in the final version.