Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields

Summary

This paper studies the anti-alias problem of grid-based static scene representation. The existing anti-alias methods like Mip-NeRF still require intensive computations, while current grid/factorization-based methods don't show good anti-alias results. This paper proposes a general idea to solve the anti-alias problem: it uses a single-scale grid as the representation while applying learnable convolution to generate multi-scale grids. Combining with scale-aware position encoding, this paper shows superior results compared to TensoRF or KPlanes, demonstrating a universal improvement for grid-based representation.

Strengths

1. This paper studies an interesting problem in an important direction. Extending the anti-alias idea from Mip-NeRF to explicit representation could contribute to the whole community, and the results are promising. I think it would be beneficial to the development of explicit representations. 2. The whole method is neat, simple, and powerful. Applying CNN on the single grid and having a scale-related positional encoding is simple and intuitive. The overall framework is neat without too many tricks and complicated tweaks. Considering it, this idea could be broadly used in many situations. 3. The paper is overall well-written and easy to follow.

Weaknesses

1. It would be more convincing to compare several baselines with multi-resolution/multi-scale designs. For example, Instant-NGP[17] uses a multi-resolution hash. Also, the original K-Plane uses multi-resolution planes. It would be helpful to show how these method works and how the proposed method compared to these baselines. 2. I am not fully convinced by the analysis of multi-scale and multi-resolution in Section 3.2. To me, the multi-scale one sounds like optimization multi-NeRF at different scales, while NeRFs from different scales are not related. Could you elaborate a little bit? It would be nice to show the experimental results of multi-resolution ideas. 3. Not necessarily a weakness. It would be good to write the method names instead of baseline 1 and 2 in Figure 3. 4. In Table 1, Mip-Kplanes's PSNR has different trends to MIP-NeRF's PSNR and mip-TensoRF's PSNR, also different from Mip-Kplanes' SSIM and LPIPS. Could you elaborate on why PSNRs are lower with lower Res?

Questions

As stated above, it would be good to show (1) more baselines, especially baselines using multi-scale/resolution methods. (2) experiments compared to multi-resolution grids.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I think the limitations discussed in this paper make sense. I agree the current method is still preliminary and having MIP-NeRF style design would be an exciting direction.

Summary

This paper proposes mip-Grid to integrate anti-aliasing techniques into grid-based representations for radiance fields. Mip-Grid generates multiple grids by applying convolution operations over the shared single-scale grid representation. Then it uses the scale-aware coordinate to retrieve the appropriate features from the generated multiple grids. This anti-aliased grid representation combines grid-based representations for faster training and Mip-NeRF-like multi-scale representations for anti-aliasing. The key idea is to use a shared single-scale grid representation and a single sampling but generate multiple grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multiple grid representations using an additional input coordinate.

Strengths

This work mitigates the aliasing artifacts when rendering images from varying camera distances, such as blurry and jaggies in the images, while enjoying fast training time. This work uses a single-scale shared grid representation and a single-sampling approach, which only introduces minimal additions to the model parameters and computational costs. The convolutions to generate multiple grids are lightweight and do not incur substantial computational and memory costs as it operates on reduced dimensions. This work shows comparable performance to mip-NeRF on multi-scale datasets while achieving significantly faster training time. This work improves the rendering performance of TensoRF and K-Planes on multi-scale datasets while introducing tolerable training time.

Weaknesses

Given the fact that Zip-NeRF is implemented on the hash-based Instant-NGP, thus it achieves fast training time despite using multisampling techniques. I will concern about the contribution of this work, as Zip-NeRF shows good performance for both anti-aliasing and training time. I would like to see the attitude of other reviewers. I believe "unbounded" scenes are the current research topics for Anti-Aliased NeRFs. Though I understand that addressing these scenarios would require other techniques, the lack of experiments of unbounded scenes (such as Mip-NeRF 360) will degree the rating of this work. Lack of visualization videos.

Questions

I would suggest including the comparison with Zip-NeRF since an unofficial PyTorch implementation of Zip-NeRF is available.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Lack of experiments of unbounded scenes (such as Mip-NeRF 360).

Summary

This work proposes a simple anti-aliasing method for NeRF representations based on factorized grids, e.g., TensoRF and K-planes. The key idea is to learn a lowpass filtering kernel

Strengths

1. The proposed method seems simple and easy to reproduce. Example implementation is also included in the supplementary. 2. The method is evaluated against baseline methods without the proposed anti-aliasing technique, and clear improvements in anti-aliasing can be seen both quantitatively and qualitatively.

Weaknesses

1. It doesn't seem clear how the multi-scale grid is used exactly in rendering; for example, are all the scales used to get a point's feature, or only a heuristically selected scale is used? 2. One thing I'm a bit unsure is that: why does the same kernel size seem to be used across all scales? Would it makes more sense to use bigger kernel size for lower-resolution grids? 3. A visualization of the kernel weights will be helpful in understanding what filtering patterns have been learned, especially the relationship of the grid scale and learned kernel weights. 4. In table 1 and table 2, best models should be highlighted to facilitate reading.

Questions

See the first and second bullet points in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The proposed method doesn't seem to be easily applicable to the multi-resolution hash grid proposed by Instant-NGP. It might be better to mention this.

Summary

The paper proposes a method to deal with the aliasing issue in factorised grid-based NeRF methods, e.g. TensoRF and K-plane, by applying convolution operations to decomposed vectors and matrices. --- **After rebuttal**: I have read authors' rebuttal and it addresses my concerns.

Strengths

* The goal of handling aliasing in grid-based NeRF is valuable and the proposed method is novel. * Source code is provided in supp mat.

Weaknesses

About evaluation: * I am interested to see what is the performance difference between using convolution operations and using a simple Gaussian? About methodology: * I got the impression that the proposed method can only handle a number of *discrete* scales after training, since the convolution kernels are optimised for each scale. If I understand correctly, that means a set of convolution kernels is optimised for a specific scale. If a NeRF is optimised for $4$ scales, for example, scale=$1, \frac{1}{2}, \frac{1}{4}$, and $\frac{1}{8}$, then during inference, it’s impossible to render an image for scale=$\frac{1}{3}$ and $\frac{1}{6}$ from this NeRF? * Following the point above, if my understanding above is correct, it would be much better to be able to render images from continuous scales.

Questions

See the weakness section above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes.

Reply to Rebuttal

Thanks for the reply and it indeed helps.

Quick clarification about 'Continuous scale coordinates'

Thanks for the response. Just double checking: is the **continuous scale coordinates** $s'$ introduced in the rebuttal stage, after submission?

Reply

I maintain my rating. Thanks to the authors for the extra details in the rebuttal, particularly the comparison between Zip-NeRF in Table 1 and the multi-scale 360 evaluation in Table 6. It would be helpful to include these in the camera-ready paper/supplement.