PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation

Summary

This paper proposed a point cloud-based representation for neural rendering of large-scale scenes. Point cloud diffusion model is designed to upsample the point cloud for better performance. The proposed method achieves state-of-the-art performance in the tested scenes. However, I still have some concerns about the implementation details and the performance comparison.

Strengths

1. The proposed method achieves state-of-the-art performance in the neural rendering of large-scale scenes. 2. Using diffusion models for point cloud upsampling is interesting and useful in the tested scenes. 3. The presentation and writing of the paper is good and the visualization comparison is clear.

Weaknesses

1. Size of the train dataset for diffusion. Diffusion model typically requires large-scale dataset for better performance. I'm curious about the number of training point cloud samples for diffusion model. 2. Reconstruction quality. The 3D reconstruction quality of MVS methods like COLMAP is not guaranteed and there can exist large holes in texture-less regions (e.g., white walls in outdoor scenarios.) I wonder the performance of the diffusion model in such situations (as the diffusion model in the manuscript is mainly designed for upsampling, not completion). 3. Comparison with Point-NeRF. Point-NeRF is an obvious baseline for this method, which should be included in the experiments. The ablations in the experiments can be seen as a variantion of Point-NeRF. However, Point-NeRF include a point grown and pruning strategy, which can also densify the point cloud. It would be better to test the performance of this strategy in large-scale scenes, which has not been explored before. 4. Inference time. The proposed method exploit an explicit reconstruction of the scene, thus the empty space can be simply skipped for better efficiency. However, the complexity of the proposed method is not reported and some implementation details are missed (e.g., the number of sampled points) 5. Background features. Mip-NeRF 360 is utilized for background rendering. However, as shown in Fig.5, the background also contributes to fore-ground regions. It would be better to include an explanation for this. 6. Others. Diffusion models are widely used for generative models and their exsiting randomness in the generated results. However, the 3D reconstruction/the neural rendering should be deterministic. I wonder the influence of the randomness in this method. Besides, I'm also curious about the failure cases of the method.

Questions

Please refer to Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to Weaknesses (especially 2. and 6.).

Summary

The paper presents a method for reconstructing large-scale scenes from multi-view images. Given the sparse point cloud computed from an MVS pipeline, the algorithm first employs a point cloud diffusion model to upsample the point cloud. The utilization of the diffusion model could effectively create a dense cloud with missing regions completed. Then, a Point-NeRF-style model is used to reconstruct the scene based on the upsampled point. Foreground and background renderings are fused via an additional fusion network, resulting in the final output. The algorithm is tested on OMMO and BlendMVS datasets. Compared to existing NeRF baselines, the proposed pipeline could limit the sampling region to the surface and achieves better reconstruction and rendering quality.

Strengths

1. Using the diffusion model for point cloud upsampling and completion is novel. While previous methods perform point cloud upsampling or completion using VAE or GAN models, this is the first paper that leverages the power of diffusion models. One related work could be Luo et al. [a], yet the proposed algorithm treats the partial input cloud as a condition and hence is also able to generate an arbitrary number of points. 2. The results are compelling. Using the upsampled point cloud as the supporting domain of NeRF, the method could reconstruct a much larger scene with finer details. Both the qualitative and quantitative experiments demonstrate the effectiveness of the proposed approach. [a] Luo, Shitong, and Wei Hu. "Diffusion probabilistic models for 3d point cloud generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Weaknesses

1. While the use of the diffusion model for point cloud upsampling is novel, its combination with the subsequent NeRF model is very straightforward and detached. The NeRF model is only a slight modification from Point-NeRF, with the addition of the background MLP, and such a representation itself is not insightful enough to form an individual contribution. 2. Some descriptions regarding the method section and experiment section are not very clear. What is the training scheme of the diffusion model? (more details in the 'Question' section). What does '... as a feature $f_i$ of each sampling point $p_i$ to equip structural information' in Line 172 mean? 3. Ablation study using other point cloud upsampling methods is missing. Given the main contribution of the paper is the diffusion-based upsampling network, it is worth investigating if a diffusion-based backbone could surpass a traditional GAN-based backbone. The study could be conducted on a simple toy dataset.

Questions

The main question arises from the training strategy as well as the model specifications of the diffusion model: 1. What is the model architecture of the denoiser? How many parameters are needed? 2. Which dataset is used to train the diffusion model? How many training samples are there? Is data augmentation applied? 3. How many points are generated from the diffusion model? 4. What is the mechanism that allows the diffusion model to complete missing regions (as shown in Fig. 5)? Are regions randomly masked out during the training stage to encourage the model to hallucinate missing contents?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not applicable.

Summary

This paper proposes an implicit neural representation for large-scale scenes with two major components, the first one is the point diffusion implicit function (PDF) which adopts diffusion process to generate dense point cloud from point cloud produced by Colmap, the second one is the background rendering which basically borrow the idea from Mip-Nerf360.

Strengths

• The paper is well organized and easy to follow. • The qualitative comparison on the fly-view dataset clearly shows the effectiveness of the proposed method on the large-scale scenes, where the rendered images are sharp in both foreground and background. • Most of nerf paper focus on small or medium scene, this paper instead provides a interesting solution to the large scale scene representation.

Weaknesses

• The effectiveness of the diffusion process, the motivation behind adding the diffusion process over colmap generated point cloud is not clear to me since the PointNerf method can generate clean and sharp foreground rendering directly with the colmap point cloud. Some explanation or visualization showing the point cloud before and after diffusion will be very helpful. • One major advantages of point-cloud based neural field is they can avoid prohibitive reconstruction time of Nerf, so a question what is the training time and rendering speed of this paper. Will the diffusion process significantly slow down the training speed? • The overall technical contribution of this paper is weak. The main component PDF shares the major design with PointNerf except applying a diffusion process over the Colmap-generated point cloud. Currently there is no enough discussion over the insight behind this design as I said before. The background rendering module also follows the existing technique from Mip-Nerf 360. Therefore I think the contribution is not enough. • In the experiment part the author said they use two dataset for evaluation but I didn’t find the result on BlendMVS

Questions

• How was the diffusion process necessary for large-scale scene representation, I do see an ablation study about that module, but I am not sure why removing the diffusion make the method totally failed but the PointNerf can works well with pointcloud directly from COLMAP. More explanation or visualization will be very helpful. • What’s the runtime of the proposed methods. • A general (but maybe not that related to the proposed method): Is there any other solutions to the background rendering? Will the environment map work for the background of the large outdoor scene.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

The authors have discussed potential negative social impacts of this paper.

Summary

In this paper, the authors propose a new approach to tackle the reconstruction of Neural Radiance Fields from large, unbounded scenes. A major limitation of current neural fields is the lack of scalability due to the number of sampling points in empty space, which is limiting the achievable scale or quality with limited compute resources. In this method, the sampling space is restricted to the surface of the object. To get an approximation of the surface of the scene, they utilize a point cloud, which first is reconstructed through MVS and upscaled to achieve a dense point cloud on the surface using a diffusion probabilistic model. Only sampling points in the near distance of the surface point cloud and the respective feature, which is constructed as a predicted feature from the neighborhood points and sampling points and forms the foreground feature together with the constructed feature. The unbounded background of the scene is modeled on a sphere as features. Foreground and background features are fused in another MLP.

Strengths

- The presented method builds on known techniques for neural rendering and generative super-resolution and combines those to solve a novel task of reconstructing large-scale unbounded scenes. I think this is a big strength of the paper, which is justified by qualitative and quantitative results and ablation studies. - The training approach shows a clear explanation of how the missing ground truth data for the point cloud can be optimized with the limited resources - The authors present their method in an understandable way, adding the necessary context and references to reproduce the work - Presented results show a clear advantage over standard NeRF techniques, and better results in areas of fine textures compared to methods designed for large-scale or unbounded scene - Most design choices are ablated and justified in separate studies

Weaknesses

- One major weakness of the proposed method is that it is required to retrain the whole model per scene. This is also mentioned by the authors in their limitations but leads to really long training times compared to sota methods. - While the result section presents all relevant quantitative results in a table, the authors do not provide an explanation for some of the outliers and rather good performances of the baselines in some scenes. - I hope I have not missed something, but there is no explanation of Fig.4. To my understanding, this provides a scene in which Mip-NeRF 360 fails completely to reconstruct the scene, but no explanation is given. Also, I am curious why it results in something like this and not only blurry results as reported in Fig.3 - One major limitation or experiment important for the specific task of large-scale, outdoor scene reconstruction is to show the boundaries of the method in terms of scene scale. Mega-NeRF is specially designed to divide a scene in multiple sections and might be able to handle larger scenes, which is not the case for this method. An additional study on the limitation of this method in terms of scene size alongside with training and rendering times would probably just strengthen the claims

Questions

- Can you provide the training and rendering times for a given resolution for your and all reference methods - The authors mentioned that the method was only evaluated on a subset provided by the authors of OMMO, but the full dataset results in the supplementary provide similar insights.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Most limitations are mentioned at the end of the paper. As discussed in the weakness section, I think an evaluation of limitation wrt scene size and computational times in a similar setting, would provide a better understanding of the capabilities of this methos.

Summary

The paper proposed a novel method for the novel view synthesis of large-scale outdoor scenes. The method combines a Point-NeRF-baed rendering stage for the foreground, as well as a background stage based on Mip-NeRF 360. To cope with the issue of overly sparse point cloud from COLMAP MVS, a point could diffusion model is trained on each individual scene from existing MVS point cloud, and used to synthesize more points. The proposed method achieves leading performance on both large and small scenes.

Strengths

* Training a scene-specific diffusion model for point cloud upsampling is a novel approach that seems to work well. * The proposed method is able to achieve better quality on a large set of real-world scenes both quantitatively and qualitatively.

Weaknesses

* The proposed method requires a diffusion model to be trained on each scene, which can require significant computation. * In terms of comparison with previous works, it is not sure if the comparison is based on a level ground, as the proposed method might require more computation, more parameter count, or both. * It is not sure if the proposed method is able to outperform methods that are based on grid data structure, such as InstantNGP, Plenoxels, etc., which tend to be very fast and scalable to large, sparse and unbounded scenes. * It is not sure if other ways of upsampling the point clouds will also work. Although the use of a scene-specific diffusion model to upsample point could is novel, it is not compared against any baselines. It might provide better insight and more helpful for future work if this contribution can be studied independently as a separate work.

Questions

* Could you elaborate more on the reason behind the failure of Mip-NeRF 360 in Figure 4? * Why is the background partially missing in Fig. 5 column 1?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Although there is a Limitations section, it only includes future works and there lacked discussions on the limitations.

Further questions

I would like to thank the authors for the rebuttal. Some further questions: 1. For the InstantNGP experiment, will the result improve with longer training? 10 seconds of training seems to be too short. 2. I wonder if you have used the NeRF++/MipNeRF360 scene contraction in the Instant NGP experiment?

Response

I appreciate the detailed rebuttal. As for the performance of Point-NeRF, have you also used background fusion for Point-NeRF (maybe a simple fusion with MipNeRF 360)? I observed that the performance of Point-NeRF is even worse than the version of yours w/o diffusion (according to the rebuttal for Reviewer yNkC).

Commet

Thank you for the extensive answers to my questions. Those provide the necessary context to understand the results and computational costs of the method and I encourage the authors to add those to the main paper Wrt. the bad case in Mip-NeRF360 I do not think it is a fair comparison for this specific case. Although Figure 4 is presenting robustness across scenes. With the lack of context provided in the attached pdf, this is not painting representative of the performance of the method. It would be good to either change the wording or add some context to the Figure description. Thank you for addressing the confusion concerning the OMMO dataset. That makes sense!