CorresNeRF: Image Correspondence Priors for Neural Radiance Fields

Summary

The submission #8302, entitled "CorresNeRF: Image Correspondence Priors for Neural Radiance Fields" proposes a novel set of losses to improve the quality of NeRF under challenging conditions. In particular, the developed strategy effectively deals with the problem of sparse images. To achieve these performances, the authors take advantage of an out-of-the-box matching strategy between pairs of images to enforce geometric constraints during the training of the implicit representation. In particular, two types of losses are proposed to improve the quality of the reconstruction, namely the reprojection loss and the depth loss. An extensive series of experiments demonstrates the relevance of these extra losses incorporated into the training. Another advantage of the proposed strategy is that it can easily be integrated into most implicit reconstruction techniques.

Strengths

- The paper is well-written and straightforward - The approach is simple and can easily be integrated into most NeRF-based approaches leading to improved results - The proposed technique is very effective under sparse view constraints - The computational overhead is very limited as matching strategies are often very fast

Weaknesses

- The approach is very simple; the losses in themselves are not really new but demonstrate very effective results. The contributions appear to be limited, but the quality of the results might justify an acceptance. For this reason, I would like to express a mixed opinion regarding the acceptance of this work. Note that a relatively similar loss (on 3D structure obtained via correspondences) is applied in "Structure-Aware NeRF without Posed Camera via Epipolar Constraint" but with less success than in this manuscript #8302. - CorressNeRF demonstrates good performance when few images are used, but it would be interesting to know the effect of these losses in more common scenarios.

Questions

- An ablation study with an increasing number of images could be interesting to analyze how the method scales and if the demonstrated improvement are also valid with a larger density of images. - Analysis with different matcher/keypoints would be a plus. - What is the effect of the density of matched points? For instance, if a very sparse SIFT matching was used, what would be the expected effect of that? - Exploring other types of loss would be interesting, for instance, some epipolar losses instead of the reprojection loss.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

I have very little to say about this paper as it is very clear and straightforward. I would like to kindly recommend additional experiments, as explained in the previous section.

Summary

The paper presents NeRF regularization method for few view NeRF.

Strengths

1. Using the state-of-the-art image matcher to regularize NeRF training is novel. 2. The paper is well-written and clear.

Weaknesses

1. The paper proposes employing the cutting-edge image matcher to enhance NeRF training. However, a similar idea was presented in Neuris [1], which also suggested using patch matching to optimize NeRF training. Intuitively, one could assume that a state-of-the-art image matcher would identify more precise correspondences than patch match, leading to superior results. However, considering that Neuris integrates additional monocular depths and surface normals, the effectiveness of combining these three methods remains uncertain. Therefore, the author appears to have overlooked an essential baseline, Neuris. It is recommended that the author carry out experimental work based on the Neuris setting rather than implementing Neuris in their own setup, which make the conclusion more convincing. 2. The pixel loss and depth loss appear to aim towards the same goal. The concept of using both has been previously explored in DSAC [2], but was later discarded in DSAC++ [3], deemed as unnecessary. While the author provides an ablation study to illustrate the effectiveness of the reprojection loss, its value remains questionable. This is primarily because, in multiview settings, the reprojection loss mirrors the depth loss, making its unique contribution uncertain. 3. Minor, The citation of UNISURF in Figure 5 seems to be wrong. [1] NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors [2] DSAC-Differentiable RANSAC for Camera Localization [3] Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC

Questions

Can the author show the proposed method is better than Neuris?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The image matcher sometimes fails when there are not enough overlap regions.

Summary

This paper proposed CorresNeRF, a method that leverages image correspondence priors to improve NeRF training on sparse input views. The correspodence matching is computed by off-the-shelf methods. The authors augue that the introduced inexpensive image correspodence priors can be used to supervise training of arbitrary NeRFs and lead to better performance / faster convergence taking sparse view inputs. Further, a robust correspondence loss is designed, including reprojection loss and depth loss baesd on correspondence priors. Overall, the method demonstrates superior reconstruction quality against baselines like VolSDF and NeuS, etc.

Strengths

The paper's main contributions are summarized as follows: - Introduction of image correspondence as a cost-effective prior to supervise the training of any NeRFs. - Design of a pipeline to obtain robust correspondences from standard methods, including automatic augmentation and filtering. - Introduce a robust correspondence loss incorporating reprojection loss and depth loss based on correspondence priors. - Extensive experiments conducted on various baselines and datasets demonstrating the method's effectiveness across different types of neural implicit representations. The authors conduct extensive experiments on various datasets, which demonstrate the effectiveness of their method. They find significant improvements in both novel view synthesis and surface reconstruction metrics. The proposed method outperforms other state-of-the-art sparse-view reconstruction methods and works well with various types of NeRF, including those with other priors.

Weaknesses

There are some limitations that should be properly discussed: - Dependence on the quality of image correspodence. The quality of the obtained image correspodence matching significantly impacts the effectiveness of the proposed approach. Less accurate correspondences can negatively affect the supervised training of NeRF, which might lead to suboptimal reconstruction results. - Performance in non-sparse scenarios. The method is focused on the advantage of using CorresNeRF in sparse-view configurations, but it doesn't mention how this method would perform in dense-view configurations. It would be interesting to show such an ablation study to verify this. - The main comparisons show VolSDF and NeuS results as baselines. However, a more adequate baseline would be SparseNeuS: SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. ECCV 2022. - The method shows 3 input views on DTU/LLFF datasets. What happen if arbitrary number of input views are given? This is soemwhat related to the 2nd point above. But it would be nice to have such experiments to better evaluate the robustness of the proposed pipeline.

Questions

Some questions: - Clarification on the image correspondence methods.The paper mentioned that image correspondences are obtained via off-the-shelf method. Could you provide more details on how they are acquired and how this (i.e. selection of the method) would affect the final reconstruction quality? - Robustness to the noise/outliers in the correspondence. An automatic augmentation and outlier removal process was designed, which seems to be key to the proposed approach. It would be useful to provide more details on the robustness of the design with some various level of noise/outliers on some scenes. - What is the minimum quality of correspondence necessary for the method to outperform traditional NeRF implementations? - Impact of loss terms on results. How did the inclusion of correspondence pixel reprojection loss and correspondence depth loss affect the final results? Could you clarify how these losses contribute to the reconstruction quality? - Extreme and failure cases. One such case I'd imagine is glossy/specular or textureless surfaces.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are discussed in the above weakness section.

Summary

The paper introduces CorresNeRF, a method that leverages image correspondence priors to improve the performance of Neural Radiance Fields (NeRF) in scenarios with sparse input views. The authors propose a plug-and-play module that incorporates correspondence priors into the training process by adding loss terms on the reprojection error and depth error of the correspondence points. They develop an adaptive algorithm for augmenting and filtering the correspondence priors to enhance their quality. The proposed method is evaluated on novel view synthesis and surface reconstruction tasks using density-based and SDF-based neural implicit representations across different datasets. The proposed CorresNeRF utilizes image correspondence priors to supervise the training of NeRF models. This approach addresses the challenge of sparse input views and enhances the performance of NeRF in reconstructing 3D geometries. The authors propose an automatic augmentation and outlier removal process for improving the quality and robustness of the correspondence priors. This process enhances the dense correspondence estimation and mitigates the effects of inaccurate correspondences. The paper formulates a correspondence loss that incorporates reprojection and depth errors based on the correspondence priors. This loss effectively guides the learning of implicit functions in NeRF models and improves their performance.

Strengths

The paper demonstrates several strengths across different dimensions: The paper introduces the concept of leveraging image correspondence priors to improve the performance of NeRF models in sparse-view scenarios. This novel approach addresses the challenge of reconstructing 3D geometries with limited input views and introduces the use of image correspondences as explicit supervision for learning implicit functions in NeRF. The combination of image correspondence priors and NeRF training is a creative and innovative approach that expands the capabilities of NeRF models. The paper addresses a significant problem in the field of 3D reconstruction and view synthesis. Sparse-view scenarios are common in real-world applications, and improving the performance of NeRF models under such conditions has practical implications. The proposed method offers a practical and effective solution by leveraging image correspondence priors, which are readily obtainable and can be computed using standard methods. The experimental results demonstrate the superiority of the proposed approach over previous methods, highlighting its potential for advancing the state-of-the-art in novel view synthesis and surface reconstruction tasks. The paper presents a well-designed methodology with clear objectives and a systematic evaluation process. The authors carefully consider the limitations of existing methods and propose solutions to overcome them. The proposed CorresNeRF method incorporates robust correspondence loss and automatic augmentation and filtering of correspondence priors, enhancing the quality and effectiveness of the training process. The experimental evaluation is thorough, encompassing various neural implicit representations and datasets, and the results demonstrate significant improvements in performance metrics. The proposed method is described in a structured manner, with detailed explanations of the augmentation and filtering process, formulation of correspondence loss, and evaluation metrics. The figures and equations further enhance the clarity of the paper, aiding in the understanding of the concepts and techniques presented.

Weaknesses

While the paper demonstrates several strengths, there are also a few areas where it could be improved: Experimental Evaluation: The paper would benefit from a more detailed analysis of the computational efficiency and resource requirements of the CorresNeRF method. Providing insights into the computational demands and resource utilization of the approach would help readers understand the practical implications and scalability of the method. Image Correspondences: While the paper introduces image correspondences as priors, it is important to acknowledge the potential challenges in estimating accurate and robust image correspondences, especially in scenarios with occluded or noisy images. Conducting a sensitivity analysis of correspondence accuracy would provide a clearer understanding of the method's performance under different conditions and shed light on its robustness and generalization capabilities. Comparison with State-of-the-Art: The paper would benefit from a more comprehensive comparison with existing state-of-the-art methods for sparse-view reconstruction, such as MVSNeRF and GeoNeRF. Providing a thorough evaluation and comparison against these methods would help establish the superiority and novelty of the proposed CorresNeRF method.

Questions

Could you provide a more detailed analysis of the computational efficiency and resource requirements of the CorresNeRF method? Specifically, it would be valuable to include information on training time, inference speed, and memory utilization to understand the practical implications and scalability of the approach. Estimating accurate and robust image correspondences can be challenging in real-world scenarios. It would be helpful to discuss the performance under inaccurate correspondence. It would be beneficial to provide a more comprehensive comparison with existing state-of-the-art methods for sparse-view reconstruction, such as MVSNeRF and GeoNeRF. Including a thorough evaluation and comparison against these methods would strengthen the justification for the superiority and novelty of the proposed CorresNeRF method.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitation is discussed.

Summary

The paper proposes an approach for sparse-view NeRF reconstruction by using image correspondences as a prior. NeRF under the sparse-view regime is overparameterized and under constrained hence requiring a prior to optimize. This paper proposes to use image correspondences that are extracted across the different views, in particular, they use DKMv3. They propose two additional loss functions based on the correspondences from the prior: the first uses the reprojection error by using the expected depth as predicted by the NeRF while the second uses a correspondence-depth based loss by finding the closest 3D points in space based on the correspondence from the prior. Experiments on novel view synthesis and surface reconstruction show the improvement of their proposed approach.

Strengths

The paper proposes to use image correspondences as a prior for sparse view nerf reconstruction which is intuitive and sound. Image correspondences as a prior is generalizable and hence a pretrained model can be used, namely DKMv3. They propose two simple, yet intuitive losses for their approach. Experiments show that the proposed method is able to perform better compared to existing baselines.

Weaknesses

The effectiveness of the method is relying on the accurate prediction of the correspondences, and it is known that correspondences can be erroneous on texture less regions, illumination changes or wide-baseline cameras. On real scenes, these issues might arise more -- e.g. on sparse scannet images as used by existing benchmarks [53, 54]. The sparse view inputs here have wide-camera baselines as opposed to forward facing scenes in LLFF. It would be more convincing if the method can also perform reasonably in such settings. Some references on sparse view NeRF: **[53]** Dense Depth Priors for Neural Radiance Fields from Sparse Input Views, CVPR '22 **[54]** SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates, CVPR '23

Questions

1. What happens when other image correspondence priors (not DKMv3) is used? It will be beneficial to see how this would affect results, for example for both another neural network based prior as well as a handcrafted prior, e.g. output of COLMAP, even if it will only give correspondences at sparse pixel locations. 2. NeRF depth is used for the correspondence pixel reprojection loss, and it is known that in the sparse regime the NeRF depth can be erroneous. Did this cause an issue in the convergence of the network? Would it have helped to add in a depth loss supervision as a prior?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have included limitations in the main paper of the submission.