Binary Radiance Fields

Summary

This paper introduces a new voxel grid radiance field representation in which the feature vectors are restricted to contain binary values. The motivation for this is to greatly reduce the storage requirements of voxel grid radiance fields. They use the straight-through estimator to allow backpropagation through the binary values. The representation combines a 3D hash grid with a triplane hash grid and uses trilinear or bilinear interpolation on the binary feature vectors. They conduct experiments on the Synthetic-NeRF, Synthetic-NSVF, and Tanks and Templs object datasets and compare against many other voxel grid radiance field methods. They show that they are able to achieve competitive rendering quality while using an order of magnitude less storage than the best-performing competitor.

Strengths

The idea is straightforward but novel to the best of my knowledge. They are able to achieve high-quality renders with a large reduction in storage requirements. It is surprising to see that a radiance field built only with binary feature vectors could perform so well. They also outperform other methods in which the radiance field is compressed as a post-process. This is probably because they are able to train the quantized model in an end-to-end manner, which is interesting to see. There is also some innovation in the idea of combining a 3D voxel grid plus a triplane representation as usually only one or the other is used, as far as I know.

Weaknesses

They are missing a reference to Variable Bitrate Neural Fields: Takikawa, T., Evans, A., Tremblay, J., Müller, T., McGuire, M., Jacobson, A., & Fidler, S. (2022, July). Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-9). However they include other compression-based methods in the evaluation, and VBNF did not provide results on the standard benchmark datasets, so I can understand why they didn't include it in the evaluation. It would have been nice to see videos in the supplemental material, which would helpful for appreciating the visual quality of the results, especially for dynamic scenes.

Questions

Is the model actually being stored as binary vectors during training and inference, or are you actually using 8-bit or larger integers (for example due to the way TinyCudaNN is implemented)? I wondered if it is actually feasible to represent binary vectors in this way using PyTorch, or if that is only a hypothetical currently.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The technical limitations appear reasonable; there is no mention of broader social impacts.

Summary

This paper proposes a new representation, binary radiance fields (BiRF), for memory-efficient novel view synthesis tasks. The representation is inspired by the binary neural network. BiRF is built upon Instant-NGP. The critical component of this representation is the binarization of real-valued feature grids, such that the resulting feature grids can store bitwise feature grids, which highly reduces the storage of NeRF models. BiRF also enhances the 3D voxel grid with three 2D plane grids, where the 2D features are incorporated to alleviate the hash collision. The training loss composes of the RGB loss and the sparsity loss. Experiments are conducted on the NeRF-synthetic dataset and NSVF dataset. Compared with state-of-the-art NeRF methods (data structure-based and compression-based methods), BiRF can obtain comparable or even higher reconstruction quality while requiring less storage. The ablation study also shows the effectiveness of introducing the 2D planes and sparsity loss.

Strengths

The paper is well written. The idea is simple but very effective and easy to implement. The insight of combining 3D feature grids and 2D plane grids is really cool. Overall, the method proposed in this paper is critical to the computer vision community (both the industrial and academia). For example, memory storage is an issue if we are reconstructing very large-scale scenes; there is also a demand for deploying NeRF models to mobile devices.

Weaknesses

The paper also mentioned a limitation is that it requires a longer time to train BiRF compared to its Instant-NGP counterpart, due to the binarization operation on real-valued feature grids. Moreover, I think the memory requirement during training can be higher than the non-binarized version since BiRF needs to maintain the temporal real-valued feature grids in addition to the binary feature grids.

Questions

The reconstruction quality of BiRF outperforms other methods by a large margin in terms of PSNR. But the reason seems not very clear since BiRF replaced the real-valued feature grids with the binary feature grids. In other words, the performance should drop (slightly) compared to the non-binary version (Instant-NGP). I think the performance gain is from the incorporation of 2D plane features. However, the authors did not provide ablations on with and without the binarization of their network architecture. I would definitely rate my score if the authors provide that.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Some minor typos are that l and L are not explained for Eq.(8), though it is obvious they're the number of grid levels. (the same issue for Eq (9)).

Summary

The paper proposes a novel approach called binary radiance fields (BiRF) for efficient storage and representation of radiance fields. BiRF utilizes binary feature encoding, where local features are encoded using binary parameters of +1 or -1. This compact encoding significantly reduces storage size and computational costs. The authors introduce a binarization-aware training scheme and extend the multi-resolution hash encoding architecture to a 3D voxel grid and orthogonal 2D plane grids. The contributions of the paper are 1. The introduction of binary radiance fields (BiRF) as a storage-efficient representation that encodes features using binary parameters. 2. A binarization-aware training scheme that effectively captures feature information and updates binary parameters during optimization. 3. The demonstration of superior reconstruction performance with minimal storage space usage, achieving impressive results in static scene reconstruction.

Strengths

1. The paper introduces a novel approach called binary radiance fields (BiRF) for representing radiance fields using binary feature encoding. This idea of employing binary parameters to represent local features in radiance fields is innovative and distinguishes it from traditional methods. The application of binarization-aware training and the extension of multi-resolution hash encoding to a hybrid structure further contribute to the originality of the approach. 2. The paper demonstrates high-quality research through rigorous experimentation and evaluation. The proposed BiRF representation outperforms state-of-the-art methods in terms of reconstruction performance while utilizing significantly less storage space. The experiments conducted on various scene datasets provide compelling evidence of the effectiveness and efficiency of the proposed approach. 3. The paper is well-written and effectively communicates the concepts and methodologies to the readers. The authors provide clear explanations of the key ideas, including the binary feature encoding, binarization-aware training scheme, and the hybrid structure of the feature grid. The organization of the paper enables easy comprehension of the research objectives, methodology, and experimental results. 4. The paper's contributions have significant implications for the field of radiance fields and 3D scene modeling. By introducing the BiRF representation, the authors address the critical challenge of storage efficiency in radiance field models, which can greatly impact practical applications. The superior reconstruction performance achieved by BiRF, coupled with its minimal storage requirements, opens up new possibilities for real-world implementation and broader accessibility of radiance fields.

Weaknesses

While the paper demonstrates several strengths, there are also a few areas where it could be improved: Experimental Evaluation: While the paper presents compelling results by reporting the model size and psnr, the paper lacks a quantitative evaluation and comparison of the inference speed with other relevant methods such as TensoRF and Instant-NGP. To what extent does the hybrid 3D and 2D feature grid architecture impact the inference speed & training speed of the proposed model? As the backward gradient to the grid is estimated & approximated, will the binary design affect the convergence speed? What is more, it seems the reported training speed of TensoRF is slower than the original paper. Results: In Figure 2, I find the the results of K-Planes on Synthetic NSVF dataset are significant worse than other methods, certain analysis of the inferior results should be discussed. It is crucial to consider and provide information about the training time and inference speed, as these factors play a significant role in assessing the effectiveness and practicality of the proposed NeRF model. Hash Collision: In original Instant-NGP, the hash collision is explicit handled as the largest gradients—those most relevant to the loss function—will dominate the optimization, the multi-scale will also alleviate it as collision is statistically unlikely to occur simultaneously at every level for a given pair of points. For this proposed BiRF, how about the situation of hash collision compared with instant-ngp? With binary code, it seems the multi-scale design will be less effective to prevent hash collision. Binarization of learnable parameter: The binarization with straight-through-estimator is a special case vector quantization or discrete representation, which has been explored in some recent NeRF research [1,2]. The paper could benefit from a more thorough discussion of these related works. Besides, there appears to be some duplication of content between lines 165-168 and lines 148-151. Streamlining these sections would improve the clarity of the paper. Reference: [1] Variable bitrate neural fields [2] General neural gauge fields

Questions

1. I would suggest the author to provide an evaluation of the inference speed with other relevant methods such as TensoRF and Instant-NGP, and analyze the effect of 3D&2D feature grid and estimated gradient. 2. It would be helpful to discuss the potential reasons for this performance gap of K-plans and provide an analysis of the inferior results. 3. How does the proposed BiRF model handle hash collision compared to Instant-NGP? 4. It would be beneficial to provide a more thorough discussion of these related works and explain the specific connections and differences between the proposed BiRF approach and the existing literature. 5. Streamlining the mentioned sections to enhance the clarity of the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is discussed in this work. There is no concern of potential societal impact.

Summary

This paper proposes a novel binary radiance fields (BiRF) which binarized the feature encoding to save memory usage of NeRF. In the experiments, the binary radiance field representation demonstrates superior reconstruction performance compared to state-of-the-art efficient radiance field models, all while requiring lower storage allocation. Notably, the proposed model achieves remarkable results in reconstructing static scenes, achieving PSNR values of 31.53 dB for Synthetic-NeRF scenes, 34.26 dB for Synthetic-NSVF scenes, and 28.02 dB for Tanks and Temples scenes. These impressive outcomes are attained using minimal storage space, with only 0.7 MB, 0.8 MB, and 0.8 MB utilized, respectively. The intention behind introducing the binary radiance field representation is to eliminate storage bottlenecks and make radiance fields more accessible for various applications.

Strengths

1. This paper is well-organized 2. Experiments are convincing and extensive.

Weaknesses

1. Only binarizing the feature encoding to save the memory usage is limited, since the acceleration rate is also import in network quantization and the inference of NeRF. 2. The proposed binarization of learnable parameter is limited of novelty. The analysis or discuss about bottleneck of NeRF quantization or binarization is lack, which is crucial.

Questions

See weakness.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

In this paper, the authors are proposing BiRF (BInary Radiance Fields), a storage-efficient representation for neural radiance fields. The technique relies on a hybrid representation that leverages explicit feature grids (both one 3D and three 2D, each at multiple resolutions) combined with density and color MLPs. To achieve high storage efficiency, the feature grids are binarized following the technique from Binarized Neural Networks. The reconstruction loss includes a sparsity inducing loss (similarly to SNeRG and Plenoxels). The main technical contributions of the paper are this specific storage-efficient representation of neural radiance fields with a matching training scheme and an array of comparisons against other methods. The authors demonstrate results through a number of quantitative evaluations on the Synthetic-NeRF, Synthetic-NSVF and Tanks&Temples datasets and against multiple baselines (fast ones: DVGO, Plenoxels, TensoRF, CCNeRF, Instant-NGP and K-Planes and compact ones: Re:NeRF, VQRF): - the proposed representation is indeed very storage efficient, with < 1 MB at acceptable quality, - it generally delivers quality reconstructions with lower storage requirements compared to either efficient or compressed alternatives, - training time remains reasonable (only behind Instant-NGP, DVGO and Plenoxels depending on the operating point). Lastly, the storage efficiency of the technique is showcased to be relevant for an application on dynamic scenes.

Strengths

Storage efficiency is an important aspect of making neural radiance fields more practical for concrete applications. While there are several neural radiance fields approaches focusing on trading rendering (and training) speed at the expense of storage efficiency and other approaches tackling storage efficiency separately by compressing a trained representation, the proposed approach addresses storage efficiency directly without necessitating a post-processing step after training and also without sacrificing quality. The approach offers competitive operating points in terms of quality v.s. storage, while offering reasonable training time. The method is conceptually simple and the paper does a solid job at presenting it, demonstrating its value through both quantitative and qualitative experiments, including a large number a baselines to compare against. The ablations are also thorough.

Weaknesses

Novelty is limited as this is essentially an application of Binarized Neural Networks to neural radiance fields approaches. As quality is one of the claims of the approach, a comparison against a few non-compressed and non-optimized baselines (e.g. original NeRF, mip-NeRF) would have made sense. The approach is exploring with good results another trade-off compared to the efficiency-oriented techniques that sacrifice storage. However, the only mention of rendering speed (at test time) are in the ablation on the sparsity loss and in the limitations section. The apparently unoptimized implementation is another weak point of the submission and a comparison of rendering speed, especially against the considered baselines would have also made sense. Minor corrections: - l.104 Pleoxels -> Plenoxels - l.135 consider the -> consider an

Questions

Following the above, would the authors be able to include a thorough comparison of rendering speed against the considered baselines? Binarized Neural Networks, which this paper is following, was using in some of their experiments stochastic binarization (as a form of regularization), is this something that has been considered? Table 3 in the supplementary material suggests increasing, have the authors tried going beyond {2^19,2^21}? Are the results plateau-ing?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Thanks for the rebuttal

The authors have answered all of my questions. I decided to mantain my rating for this paper.