CRAYM: Neural Field Optimization via Camera RAY Matching

Summary

The manuscript #3263 entitled "CRAYM: Neural Field Optimization via Camera RAY Matching" proposes a novel uncalibrated NeRF strategy based on prior keypoints matching across images. Specifically, the authors propose two novelties to improve the quality of the reconstruction and the pose estimation of the cameras: 1) Enriched ray features using surrounding rays sampled around the keypoints and 2) a ray matching index which can be used to re-weight the color regression part, leading to better robustness to occlusions. The proposed technique has been evaluated across various standard datasets and against meaningful NeRF-like algorithms.

Strengths

- The idea of separating key rays and auxiliary rays is interesting and meaningful. - Numerous and conclusive results. - Assessment on a large number of datasets. - Good ablation study underlying the benefit of each novelty.

Weaknesses

- The robustness of the approach against outlier matches is not evaluated. Introducing artificial outliers (wrongly matched keypoints) into the dataset to assess how well the technique can handle mismatches would be of some interest. *Question*: Would the matched rays consistency and the epipolar geometry compensate for that? Or would the training diverge? - As stated in the literature review of this manuscript, other approaches taking advantage of the epipolar geometry and prior matching have already been designed in this context. I have difficulty understanding what is significantly different with this work apart from the sampling of additional surrounding rays and the matching ray index used to weight color prediction using image pairs. These two novelties seem rather incremental, but they nonetheless lead to strongly improved results. *Question*: I assume that the other keypoints-based approaches are not "self-calibrated". Is the proposed technique the first "keypoint-based" calibration-free NeRF? If it is not the case, it would be meaningful to compare against such techniques too. - Adding surrounding rays around a key ray appears to be quite effective; however, the sampling of auxiliary rays is not well described in the paper. *Question*: How are the rays sampled? - The initialization of the pose lacks details. *Question*: What is the effect of the pose initialization on the result? - The intrinsic parameters of the camera could additionally be optimized. *Question*: Just out of curiosity, have you conducted such an experiment? - In equation (4), it seems that the proposed solution considers only pairs of images. *Question*: How are those pairs selected? The proposed approach is inspired by existing techniques integrating matched keypoints (like using the epipolar loss) and other techniques, such as NeuS. - The loss function contains many regularization factors. *Question*: Is the final loss hard to balance? Overall, the paper is interesting and proposes a few contributions that seem to lead to strongly improved results. Moreover, the approach has been evaluated on various standard datasets and against representative methods. However, this novel approach remains relatively incremental, and many points remain to be clarified regarding the robustness of the technique. For all the above-mentioned reasons, I would like to issue a rather mixed opinion regarding the acceptance of this work for this conference.

Questions

The questions are integrated in the Weaknesses part.

Rating

5

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

The limitations of the paper may not have been entirely investigated, specifically in terms of robustness. For instance, the influence of the initial pose and outliers has not been demonstrated.

Summary

This paper presents a new technique called camera ray matching, which is integrated into the joint optimization of camera poses and a neural field. The method utilizes an uncalibrated set of images as input, incorporating photometric and geometric constraints through key points and key rays matching, with the aim of enhancing the quality of novel view rendering and 3D surface reconstruction. The approach comprises two simple modules and is implemented using grid-based representation (iNGP). Photometric experiments were exclusively compared with MLP-based methods, specifically NeRF-like, on the synthetic dataset of the vanilla NeRF, while geometric experiments demonstrate some positive results. The authors provide additional results in the appendix.

Strengths

This work is a positive extension to the field of neural reconstruction (like NeRF and SDF) under the setting of images captured with noisy poses. Authors make efforts to simultaneously solve the problems involving the camera pose, detailed renderings, and accurate surface reconstruction. Experiments show good results.

Weaknesses

Too many factors are taken into account in the writing simultaneously, which leads to a lack of clear theme or a clear academic or technical problem to be addressed in this paper. The work appears to build incrementally upon previous research and offers limited novelty. The so-called Epipolar loss and Point-alignment loss are actually based on Bundle Adjustment (BA), using key points matching, which has been previously applied in the optimization of neural reconstruction in works such as SCNeRF, BARF, L2G, and Level2sfm. The proposed two modules do not bring significant innovation. It is also confusing that this work is implemented using a grid-based representation (i.e., iNGP), while the compared methods are implemented using MLP-based representation, which does not allow for a precise and fair comparison. I suggest that the authors refer to ZipNeRF for guidance on how to formulate research problems and conduct appropriate comparisons.

Questions

Based on the mentioned shortcomings, I have several questions: 1. What is the main research problem addressed in this paper? It appears that the paper intends to address three problems simultaneously: camera pose, fine-detailed rendering, and accurate surface reconstruction. However, if the goal is to solve the camera pose problem, it would be beneficial to present more results related to the accurate regression of camera poses before discussing high-fidelity results. Could you provide additional results about the accurate regression of camera poses? 2. If the primary focus is on the issue of high-fidelity rendering under noisy camera poses, the experimental results do not offer strong conviction. It would be beneficial to showcase more experimental results on different types of datasets, similar to how BARF, SPARF, and L2G-NeRF were compared on the LLFF dataset, with an emphasis on trajectory of pose optimization and high-fidelity renderings. 3. Initialization of camera poses is a sensitive issue in joint optimization. Have you attempted to change the initial camera poses or initialize poses using COLMAP to test the robustness of pose regression? 4. This work is implemented using grid-based representation, while the compared methods are implemented using MLP-based representation, leading to an imprecise and unfair comparison. I am curious about the total number of iterations each method was trained for. BARF and L2G-NeRF trained all models for 200K iterations, and SPARF trained models for 100K iterations. It seems likely that this paper trained for significantly fewer iterations (perhaps 10k iterations) due to the inherent faster convergence of the grid-based representation. It would be fairer to re-implement this paper using MLP-based representation and train for a duration comparable to other methods, then make comparisons. Additional experiments are needed to support these claims. 5. The comparison under sparse-view conditions lacks precision and distinction. SPARF evaluated on the DTU dataset with only 3 views, while you used 48 images in your setting, although you have reported results for sparse input (3 views) on the LEGO data, it may not be sufficiently convincing. 6. Since Neus and PET-NeuS have compared the surface reconstructions on the DTU dataset, I would appreciate the inclusion of visual results. Additional ablation studies on 3D surface reconstruction using your modules on the DTU dataset would enhance the paper. 7. Considering the integration of multi-resolution hash encodings into a neural surface representation, I recommend that the authors compare with Neus2. Neus2 has compared their work with Instant-NGP and Neus, showcasing fast training and detailed surface reconstruction. If the goal is to demonstrate the superiority of 3D surface reconstruction, it is most appropriate to consider Neus2 in the comparison.

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors have identified a limitation where the meshes extracted from the constructed SDFs may still contain messy inner structures over invisible areas. I recommend that the authors explore the possibility of finding the SDFs of surface points instead of the SDFs of each sampled point along rays. This would involve assessing whether the depth accumulated by all points along the rays is more accurate than the output surface generated by all sampled points.

Summary

This work suggests a novel neural representation and training scheme that jointly solves for the scene representation and the multi-view camera localization. It is done using several new ideas that generalize existing NeRF based methods. The representation itself is a combination of a geometry-network, which predicts a signed-distance-function (SDF) and a feature vector, that are fed into the texture-network that predicts the usual color and density values. The main key novelty, is that the optimization is done over matching rays, obtained from matching keypoints using a pretrained network. The standard photometric loss function is extended to incorporate an epipolar loss (that constrains the camera positions) and a point-alignment loss that ensures the ray intersect at the predicted depth estimates along the rays. Another strong addition, is the use of 'auxiliary' rays around each matched pair of rays, from which features are fused to produce a more robust representation, that can aid the optimization under errors in matching and camera poses. Extensive experiments demonstrate the importance of each component and the strong performance of their combination.

Strengths

* The paper presents an extension of the NeRF framework, based on several novel and interesting additions that are framed in a single pipeline. The experimental results show that these contributions work well together and yield new state-of-the-art results, across the board. * One promising idea, in my view, is the joint optimization of both geometry and texture networks, which clearly complement eachother and are helpful in obtaining stronger and more accurate constraints on the scene understanding (as opposed to most NeRF pipelines that focus on image reconstruction and are less accurate for 3D reconstruction). * The other strong idea, is the joint optimization of matching rays, once again - imposing consistency contraints (on both camera and surface locations) that were not previously exploited to such an extent in prior work. * The paper is well written and the contributions are very clearly highlighted, while the understanding of the conventional parts is left for the reader (which is mostly fine).

Weaknesses

* Reproducibility - I believe that many details are missing (including from the appendix) for one to be able to implement the proposed method. For example: * What are the settings of the preprocessing SuperPoint and SuperGlue matching? What is the typical match density? * How are the auxiliary rays sampled? How many and under which distribution? * What is the function g in Eq 2 that fuses the key and auxiliary features? * What are the balance weights in the final loss (Eq. 7)? * How are poses initialized? * Complexity - There is no discussion what so ever about the impact of the suggested changes on memory and runtime complexity, but at traning and in inference. * Qualitative results are relatively limited. * Synthesized images are all very small, so it is difficult to appreciate the fidelity. * No depth images are shown * No examples of key and auxiliary point matches are shown (over entire images)

Questions

* Is the training done only on matching key points? If so, what happens if correct matching keypoints do not 'cover' the space adequately? * How sensitive is the method to the quality and density of the 2D matches? It reportedly worked well on sparse image sets, which is somewhat surprising. * What are the runtimes (training and inference) compared to some baseline methods? * How different is the use of auxiliary rays compared to sampling from conical frustums as in Mip-NeRF?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Adequately discussed.

Summary

This paper introduces Camera Ray Matching for optimizing camera poses and neural fields from multi-view images. The optimized feature volume supports novel view synthesis and 3D geometry reconstruction by probing camera rays, which carry both geometric and photometric information. CRAYM claims to improves efficiency and accuracy by focusing on keypoints and integrating multi-view consistencies, enhancing both geometric reconstruction and photorealistic rendering. The method shows result in NVS and geometry reconstruction compared to baseline methods.

Strengths

- The paper is well-structured and easy to follow.

Weaknesses

- Experiments were only conducted on NeRF-synthetic datasets and not on LLFF datasets. - Comparison is made with older baseline methods (e.g., SPARF, BARF, L2G) which are more than 2 years old. It’s recommended to include more recent methods such as NoPe-NeRF and BAA-NGP. - It is suggested that the authors perform Neural Image Alignment to enhance the evaluation.

Questions

Figure 5 and Table 3 appear to have a significant overlap in the information they present.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

YES