DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D observations in outdoor autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs with limited view overlap, and is trained self-supervised with differentiable rendering to reconstruct RGB, depth, or feature images. Our first insight is to exploit per-scene optimized Neural Radiance Fields (NeRFs) by generating dense depth and virtual camera targets from them, which helps our model to learn enhanced 3D geometry from sparse non-overlapping image inputs. Second, to learn a semantically rich 3D representation, we propose distilling features from pre-trained 2D foundation models, such as CLIP or DINOv2, thereby enabling various downstream tasks without the need for costly 3D human annotations. To leverage these two insights, we introduce a novel model architecture with a two-stage lift-splat-shoot encoder and a parameterized sparse hierarchical voxel representation. Experimental results on the NuScenes and Waymo NOTR datasets demonstrate that DistillNeRF significantly outperforms existing comparable state-of-the-art self-supervised methods for scene reconstruction, novel view synthesis, and depth estimation; and it allows for competitive zero-shot 3D semantic occupancy prediction, as well as open-world scene understanding through distilled foundation model features. Demos and code will be available at https://distillnerf.github.io/.
Paper
Similar papers
Peer review
Summary
The paper introduces a self-supervised learning framework named DistillNeRF for understanding 3D environments from limited 2D observations. This framework is designed for autonomous driving and leverages per-scene optimized Neural Radiance Fields (NeRFs) and features distilled from pre-trained 2D foundation models such as CLIP and DINO. The model predicts rich 3D feature volumes from single-frame multi-view camera inputs, supporting various downstream tasks like scene reconstruction, novel view synthesis, and zero-shot 3D semantic occupancy prediction. Experimental results on the NuScenes dataset demonstrate the effectiveness of DistillNeRF over existing methods.
Strengths
1. The paper is interesting to read and simple to follow. The methodology is robust, combining offline per-scene NeRF training with a distillation stage that generalizes across scenes. 2. The paper is well-structured and clearly explains the methodology, including detailed descriptions of the model architecture, training process, and experiments. Figures and tables effectively illustrate the key concepts and results.
Weaknesses
1. Computational Complexity and Inference Speed: It would be beneficial for the authors to include the training time for both per-scene EmerNeRF and DistillNeRF, as well as the inference speed of DistillNeRF. This will help to better assess the efficiency of the proposed method. 2. Results on Other Datasets: Currently, the experiments are conducted only on the nuScenes dataset, whereas EmerNeRF has been tested on the Waymo dataset as well. Including experiments on the Waymo dataset would strengthen the claims regarding the generalizability of the method.
Questions
See the "Weakness" section. 1. Since the nerf-based method requires extra training time, is it possible to extend this method to support 3DGS distillation?
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Limitations
See the "Weaknesses" section. Additionally, please note that the final score for this study is not solely determined by the peer reviewers' discussion. If the authors can address my main concerns, I would be willing to raise the score.
Response to reviewer's comment
Dear reviewer, first we would like to extend our sincere gratitude for taking the time to read through the feedback and engage in the discussion for our paper. We greatly value your precious time/efforts that you've dedicated to contributing to the conference and the broader community. We would like to clarify the point mentioned by R-WxYD, who stated that “this feels like writing the paper after the publication deadline”. However, as mentioned in our new response to R-WxYD, we are currently at the rebuttal stage rather than the publishing stage. We firmly believe, “reviewers provide constructive comments and authors improve the paper accordingly”, is one key reason to set up the rebuttal stage, and one important part of the review cycle where reviewers and authors work together to present greater works and contributions to the community. We kindly refer you to the NeurIPS reviewer guidelines, which emphasize the importance of this stage. Since you are taking reviewer WxYD's comments into your assessment, we would also gently suggest you read our responses to R-WxYD, as it could provide additional contexts that could be helpful to your evaluation. Thank you again for your time and thoughtful consideration.
Summary
This paper presents a method for 3d understanding from 2d observations for autonomous driving. The main technical contribution is a feed-forward model, which is trained by distilling RGB and depth from a per-scene optimized NeRF model. The proposed model predicts 3d feature volumes that enable volumetric rendering similar to NeRFs. Experimental results on NuScenes show improved performance compared to recent comparable generalizable methods.
Strengths
- The model is generalizable, meaning that no per-scene optimization is required at inference. Compared to recent comparable generalizable methods [18,19], the results for reconstruction, novel view synthesis and depth estimation are improved. - The authors propose a simple methodology to distill depth and RGB renderings from EmerNeRF, which requires per-scene optimization. The distillation process improves the results. - The method is extensively evaluated on NuScene, and achieves very good results for multiple tasks, namely novel view synthesis, depth estimation and also 3d occupancy prediction.
Weaknesses
- One of the main reasons for having this generalizable formulation instead of just using e.g. EmerNeRF is that it is faster, yet inference time is not discussed in the paper. Can the authors report this in the rebuttal? For Table 1, 2 and 3 inference time should be reported for all methods. - The ablations are unclear. What exactly is meant by “Ours” in Table 1, 2 and 3? Is it the model without depth distillation, param space and visual cam distillation? Why is not the model that combines all components like “Ours (+ Depth Distillation + Param Space + Virtual Cam Distillation)” tested? Since “Ours” performs worse than “Ours (+ X)” it looks like it is not tested. Why not test “Ours (full)” and e.g. “Ours (- Depth Distillation)”, more like a standard ablation study? - The part about distilling foundation models is incomplete. There are some simple baselines missing, e.g. rendering RGB and then computing CLIP/DINO from the RGB renderings. How much slower is it and how does the reconstruction accuracy compare? It is also unclear why the foundation model reconstruction is only reported for the model variant “Ours (+ Depth Distillation)” in Table 1 and not the others.
Questions
Please see weaknesses
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
This is adequately addressed
Summary
Appears to present a method for online 2D feature distillation in an autonomous driving configuration. Method appears to use some kind of pre-trained depth prediction network to build a frustum aligned grid of 2D features, which are somehow rasterized into a canonical sparse volumetric grid. There appears to be some use of networks ("embedding") to do inference on these features, which allow for 3D semantic segmentation and occupancy prediction. Method presents novel view synthesis, depth prediction and occupancy prediction metrics on the nuScenes dataset.
Strengths
Paper claims to be the first to achieve online distillation of 2D features into 3D in the autonomous driving domain. Presents some SotA metrics on novel view synthesis
Weaknesses
The papers greatest weakeness is the lack of enough detail descrbing the method to reproduce it, and therefore to be able to properly critique it. Distilling 2D features into 3D via Neural Radiance Fields is not a new idea (“LERF” , and “FeatureNeRF" are cited. “Decomposing NeRF for Editing via Feature Field Distillation”, NeurIPS 2020 is not cited), yet is given too much attention. The real contribution should be as to how this is done in real-time, but critical details as to how 2D features are lifted into 3D via depth predictions, are missing, how view frustums are combined into a canonnical 3D grid are missing, and how features are combined and "embedded" (suggesting the use of a neural network) missing. (See "Questions below for further deails"). I would have thought the use of sparse grids would more easily allow unbounded spatial discretization and the MipNeRF-360 style space contraction unnecessary ("Neural Field Parameterization"). This design choice appears to be unmotivated and un-ablated. Precise (non-verbal) definitions of all terms in the final loss Eq (4). appear to be missing. Method is presented as an online method, but provides no metrics indicating real-time performance.
Questions
#124 “depth feature map is further embedded” - how do you perform this embedding? This would suggest you are using a neural network of some sort. #127 “To this end, we first aggregate the frustum to predict a raw depth for each pixel, and then sample fine-grained depth candidates centered around the raw depth. Both depths are trained end-to-end with our model.” You provide equations for compositing (1) and expected depth (2) (which are relatively trivial and could probably be omitted), but exactly how you do the remainder appears to be missing. #130 “the frustum is designed to contain the density value”. This would be the place to describe this design. #136 “The depth feature map” It is unclear what you mean by “depth features”. Are intermediate features used to predict depth? Bucketized depth probabilities? This might reference details that are buried inside “Depth Anything”, but not apparent on a cursory read of it. #145 Unclear what you mean by “sparse quantization”. Quantization usually means discretizing the floating point value to a finite set of values. Is this what you mean? Also, what spatial interpolation and sampling technique are you using to sample from the frustum to the global grid? #151 What do you mean by “two octrees with different quantization granularities,” An octree is by definition multi-resolution. Are you simply using a sparse grid? Do you have two sparse grids? If using an octree, please describe at least how many levels are in it. Further saying you apply “sparse convolutions” makes me think you actually mean sparse grid, because a sparse multi-resolution convolution on an octree is exceedingly difficult to implement.
Rating
3
Confidence
3
Soundness
1
Presentation
1
Contribution
1
Limitations
Briefly mentions limitations in the final paragraph of "Conclusions". Though they state #334 "sparse voxel representation naturally trades off rendering efficiency for dense scene representation", no actual performance metrics appear to be given. Societal impact is properly addressed.
Response to Reviewer WxYD
We thank you for the prompt response. > Q1: "lack of enough detail descrbing the method to reproduce it"... - First, to clarify, we firmly believe that our original submission has already described the method in sufficient detail, that is already up to the standards of the field. This is also reflected by the fact that, other reviewers unanimously rated the presentation as “3 good”, and explicitly mentioned “The paper is well-structured and clearly explains the methodology, including detailed descriptions of the model architecture, training process, and experiments” and “The proposed method exhibits good soundness, with the technical details of each component well elaborated”. - In such contexts, you raised questions such as the definitions of feature maps and embeddings. We firmly believe these are preliminary knowledge in deep learning/vision, but we respect/appreciate your unique angle, and were happy to respond to them one by one in the rebuttal phase. - Third, the reviewer suggested including every fine-to-the-ground detail of the paper, for which we even wrote down the mathematical equation of L1 loss. This is not actionable: today’s AI fields are moving very fast and have grown to a level of complexity that every paper would rely on previous foundations. It has been a common writing paradigm for papers to 1) mostly emphasize the core novelty of the paper, since that’s the key message delivered to the community; 2) leave the commonly used techniques or implementation details to references, appendix, supplementary material, and ultimately the source code. We provide code not as a replacement for a clear description, but as an additional resource to aid easy reproducibility, which is common practice in the field. For example, all SOTA methods compared in our paper follow this paradigm (EmerNeRF, UniPAD, SelfOcc, OccNeRF, SimpleOcc) - Based on the points above, we firmly believe that the reviewer’s concerns of clarification and reproducibility are already well addressed. > does not sound like you intend to include these losses in the main text. - As in our rebuttal, we explicitly said “will incorporate these details into the paper”. We kindly request you carefully read our response, and make informed comments. > I appreciate the extra descriptions of the method. But these are all descriptions that should have been included in the paper. This feels like writing the paper after the publication deadline. As do the extra timing ablations that should have been included in the paper. - Note that we are at the rebuttal stage instead of the publishing stage. We firmly believe, “reviewers provide constructive comments and authors improve the paper accordingly”, is one key reason to set up the rebuttal stage, and one important part of the process where reviewers and authors work together to present greater works and contributions to the community. Please also see the NeurIPS reviewer guidelines. It’s a pity to hear that you do not recognize the responses and improvements made during the rebuttal, even if they well address your concerns. > "... [we] ... also offline-trained NeRFs" I'm unclear what you are trying to say. Are you trying to say that you distilled features offline? - This is the key/core method of our paper, namely distillation from offline-trained NeRFs, which is clearly indicated by the title of our paper (DistillNeRF), and extensively elaborated/evaluated throughout the paper (Intro/abs, Fig.1, Sec 3.2, Table 1/2/3). Again, we kindly request you read our paper carefully and make informed comments. > I have always seen "quantization" refer to the quantization of floating point values, and not the sparsity structure of the grid. Can you provide a reference that actually uses this term the same way? - In 3D vision, voxel quantization, or sometimes called voxelization, is a basic operation widely used [1,2,3,4,5,6,7,8]. [1] introduced “Commonly, point clouds are first quantized in a process known as voxelization, with the resulting voxel grid being used as input to 3D CNNs”. Also, see Sec.3.1 in [2] for a whole section of descriptions. Fig.4 in [2] and Fig.1 in [3] also show excellent illustrations for dense and sparse quantization, respectively. > I also have a hard time accepting 2fps at a very low resolution of 228x128 as a "realtime" or "online" method - First, we use the term “online model” in line with the literature, referring to reconstructing the scene with one model forward pass [9,10,11]. This contrasts with “offline" NeRFs, which obtain a single scene representation through dedicated optimization. Our method renders 6 images in 0.486 seconds, compared to an offline NeRF like EmerNeRF, which takes 1.5–2.5 hours to achieve the same. - Second, we need to point out that our model generates 6 images with 0.486s, showing 12fps instead of 2fps. The pure rendering takes 0.127s with 47fps. Again, please read our response carefully and make informed comments.
Reference in our response
[1] Zhang, Chris, Wenjie Luo, and Raquel Urtasun. "Efficient convolutions for real-time semantic segmentation of 3d point clouds." 2018 International Conference on 3D Vision (3DV). IEEE, 2018. [2] Qian, R., Garg, D., Wang, Y., You, Y., Belongie, S., Hariharan, B., Campbell, M., Weinberger, K.Q. and Chao, W.L., 2020. End-to-end pseudo-lidar for image-based 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5881-5890). [3] Huang, Lila, et al. "Octsqueeze: Octree-structured entropy model for lidar compression." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. [4] Chen, Xiaozhi, et al. "Multi-view 3d object detection network for autonomous driving." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017. [5] Ku, Jason, et al. "Joint 3d proposal generation and object detection from view aggregation." 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018. [6] Liang, Ming, et al. "Deep continuous fusion for multi-sensor 3d object detection." Proceedings of the European conference on computer vision (ECCV). 2018. [7] Yang, Bin, Wenjie Luo, and Raquel Urtasun. "Pixor: Real-time 3d object detection from point clouds." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. [8] Zhou, Yin, and Oncel Tuzel. "Voxelnet: End-to-end learning for point cloud based 3d object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. [9] Yu, A., Ye, V., Tancik, M. and Kanazawa, A., 2021. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4578-4587). [10] Charatan, D., Li, S.L., Tagliasacchi, A. and Sitzmann, V., 2024. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19457-19467). [11] Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N. and Funkhouser, T., 2021. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4690-4699).
I thank the authors for their response, but not their combative tone ("It’s a pity to hear that you do not...", "but we respect/appreciate your unique angle...", "We kindly request you carefully read our response, and make informed comments"). There may be a cognitive bias to apply a perception of laziness and mis-understanding to the reader, but I would suggest that a more productive tone would be to ask if what your are intending to communicate is actually what you are communicating? As a meta-example, you state "[you] will incorporate these details into the paper". Yet closer to your response on loss terms you further state: "Due to the character limit, we briefly introduce them here, and refer to the code for full details." This reads as a) an excuse as to why these details weren't initially included, and b) an indication that the source code will be the documentation for these losses. There is no statement that you intend to include these loss terms in the paper. The wording "brielfy introduce them here" is ambiguous. What does "here" mean? The rebuttal? or the paper? A clearer response would have been: "We agree the loss equations are essential details originally omitted due to space constraints. We will amend our submission to include the terms listed below". Ultimately, you state "we firmly believe that our original submission has already described the method in sufficient detail, that is already up to the standards of the field." Which I disagree with (with all due respect to my peer reviewers who may feel the opposite). I do not hear the authors stating unambigiously they intend to clarify their presentation.
Response to reviewer's comment
Dear reviewer, thanks for the reply. We want to clarify, please do not get us wrong, our response is fully respectful to the reviewer and the reviewer's efforts, just like the word literally means "respect/appreciate", "kindly", and just as how we respond to reviewer's questions one by one in detail during the rebuttal stage, such as explaining the definition of feature maps, embeddings, voxel quantizations, write down the mathematical definition of l1 loss to address the reviewer’s question, and introduce the common practice for writing and code sharing in the community. Apologies if any of these comments look improper from some angles, that is not what we mean. We will incorporate the clarifications into our paper, just as we will incorporate every other reviewer's suggestions into our paper, and sincerely appreciate every reviewer's great suggestions/perspectives to improve our paper and contribution to the conference and community. Specifically, for the losses, we reassure the reviewer that we will update our paper to include them, just as how we responded to you in detail during our rebuttal. Finally, we thank the reviewer again for the precious time and effort in reviewing our work, and engaging in the discussion stage.
Summary
This work aims to enhance the understanding of 3D environments from limited 2D observations in autonomous driving scenarios. It achieves this by proposing a new generalizable NeRF pipeline, trained using distillation from per-scene NeRFs and foundation models. This pipeline can transform input RGB images into 3D feature volumes that can be decoded into foundation model features at inference time in a feed-forward manner. Extensive experiments validate that the proposed method outperforms the baselines across various 3D tasks.
Strengths
1. The proposed method exhibits good soundness, with the technical details of each component well elaborated. 2. The proposed method outperforms the baseline methods across different tasks and achieves results that are on par with per-scene optimization methods.
Weaknesses
1. The new insights delivered by this work to the community are somewhat unclear. Specifically, this work combines various techniques, some of which have been partially employed in previous image-based rendering pipelines, into the proposed framework. However, it is unclear which techniques are particularly useful for the target autonomous driving scenario. If the authors aim to address the challenges specific to autonomous driving scenes, they should explicitly formulate these challenges and provide an analysis of which design is particularly useful for addressing each challenge. Although the detailed techniques may not be new, this analysis can significantly benefit the community when addressing similar scenes. 2. The differences between the components in the proposed pipeline and other image-based rendering pipelines need more clarification. Specifically, what are the key differences between the proposed method and the combination of GeoNeRF and FeatureNeRF? In my understanding, the key differences are the distillation from per-scene NeRF due to the lack of accurate depth and the adoption of multi-view fusion. 3. As a follow-up to point 2, I wonder what the advantages of adopting multi-view fusion plus a voxel grid are compared to 3D-to-2D feature projection for each sampled point along the ray in previous generalizable NeRFs. 4. Although this work mentions the real-time processing demand, the efficiency aspect of the proposed pipeline is not measured or analyzed. 5. I wonder whether the proposed method can be applied to common NeRF benchmarks, in addition to NuScenes, while still achieving leading generalizable reconstruction performance compared to other generalizable NeRF variants like IBRNet, GNT, and GeoNeRF.
Questions
My questions have been included in the weakness section. I'm willing to adjust my scores if my concerns are properly addressed.
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
This work does not suffer from notable negative societal impacts.
Settings for the under-training ablation studies.
| Two-stage LSS | Pretrained encoder | Density complement | Decoder | Metrics | |:-------------:|:------------------:|:------------------:|:-------:|:-------:| | ✗ | ✓ | ✓ | ✓ | | | ✓ | ✗ | ✓ | ✓ | | | ✓ | ✓ | ✗ | ✓ | | | ✓ | ✓ | ✓ | ✗ | | | ✓ | ✓ | ✓ | ✓ | |
Update on the requested ablation studies
Dear reviewer, may we first extend heartfelt gratitude for spending the time reviewing and engaging in the discussion stage, it never goes unappreciated. As promised, now we are happy to present the ablation studies requested, as in the table below. | Density complement | Decoder | Pretrained encoder | Two-stage LSS | Depth Distillation | PSNR | SSIM | |:------------------:|:-------:|:------------------:|:-------------:|:------------------:|:-----:|:-----:| | ✗ | ✓ | ✓ | ✓ | ✓ | 22.76 | 0.669 | | ✓ | ✗ | ✓ | ✓ | ✓ | 25.34 | 0.839 | | ✓ | ✓ | ✗ | ✓ | ✓ | 21.35 | 0.536 | | ✓ | ✓ | ✓ | ✗ | ✓ | 27.40 | 0.859 | | ✓ | ✓ | ✓ | ✓ | ✗ | 28.01 | 0.872 | | ✓ | ✓ | ✓ | ✓ | ✓ | 30.11 | 0.917 | Specifically, we ablated key components of our sparse voxel representation such as density complement, decoder, pre-trained 2D encoder, and the two-stage LSS. We remove one component each time to ablate its effect. In the last row, we also further add the depth distillation from offline NeRF, which represents the best performance of our full model. - No density complement: we observe a significant drop of PSNR from 28.01 to 22.76, demonstrating the importance of better coordination between low-level and high-level sparse voxels. - No decoder: we see a decent drop of PSNR from 28.01 to 25.76, showing the effectiveness of using the decoder for robustness to noises in rendered features. - No pre-trained 2D encoder: we see a significant drop of PSNR from 28.01 to 21.35, which is expected since using pre-trained 2D encoder has been a commonly acknowledged approach in the field, which SOTA methods such as UniPAD and SelfOcc all adopt. - No two-stage LSS: we observe a slight drop of PSNR from 28.01 to 27.40. - Add depth distillation from offline NeRF: we observe the jump of PSNR from 28.01 to 30.11. Note that every ablation above outperforms the SOTA methods UniPAD and SelfOcc, which have PSNR of 19.44 and 20.67 respectively. Among the above techniques, 1) density complement, 2) rendering decoder, 3) two-stage LSS, and 4) distillation from offline NeRF are the novel and unique methods proposed by our paper. With these ablations and analysis for each technique, we believe the key message and contribution of our paper to the community are much clearer. Finally, the authors want to take a moment to acknowledge your strong expertise and foundations in the field. Your comments are among the deepest and the most constructive, which effectively helps us improve our paper. We acknowledge your precious time spent on the review and discussion.
I thank the authors for their rebuttal. I remain very negative over the "lack of enough detail descrbing the method to reproduce it". I appreciate the authors inclusion of the source code, but there's no reason a well written Methods section _couldn't_ have included include all salient details so that I wouldn't _have to_ look at the source code. A paper should do the work of digesting the method for the reader instead of forcing them to go a level deeper and dissect the source code. Source code inclusion should only be for reproducibility purposes, and not as the primary documentation of the method. Furthermore, the authors rebuttal of "Due to the character limit, we briefly introduce them here, and refer to the code for full details." does not sound like you intend to include these losses in the main text. I appreciate the extra descriptions of the method. But these are all descriptions that should have been included in the paper. This feels like writing the paper after the publication deadline. As do the extra timing ablations that should have been included in the paper. I'm holding up the terminology of "sparse quantization" as unclear writing. I have always seen "quantization" refer to the quantization of floating point values, and not the sparsity structure of the grid. Can you provide a reference that actually uses this term the same way? I also have a hard time accepting 2fps at a very low resolution of 228x128 as a "realtime" or "online" method "... [we] ... also offline-trained NeRFs" I'm unclear what you are trying to say. Are you trying to say that you distilled features offline? Again, the language is still unclear. I appreciate the authors rebuttal but I maintain my rating as "reject".
Thanks to the author for providing the rebuttal. However, I agree with Reviewer WxYD that this work needs to be improved by including the materials in the rebuttal. Considering this point, I tend to lower my rating.
Thank you to the authors for providing the detailed response. i will keep my rating and listen to our reviewers' opinions.
I thank the authors for providing answers to all my questions. I keep my rating as weak accept.
Decision
Accept (poster)