GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields

Summary

This paper presents GeoNLF, a new approach to pose-free large-scale point-cloud registration (PCR) and LiDAR novel view synthesis (NVS). To address the limitation of both ICP-based and NeRF-based methods, the authors proposed a hybrid approach involving both robust ICP and global optimisation of NeRF. Experiments show the proposed method achieved better pose estimation and NVS results compared to previous SOTA methods.

Strengths

1. The paper is well-written and easy to follow. 2. The combination of geometric-based ICP and direct NeRF optimisation is novel and the explanation is very clear. 3. Fig. 3 and Fig. 4 are very informative in showing how the proposed geometric optimisation and selective re-weighting helps the optimisation process. 4. The experimental results are promising.

Weaknesses

1. The proposed approach consists of so many building blocks which makes it a complicated system to work. Besides, from Tab. 3, it seems that except the geometric optimizer which contributes a lot to the final result, all other components seem to only produce very small improvement. 2. How long does the optimisation take? It would be good to show some run-time comparison and analysis.

Questions

Please see the Weakness.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

In this submission the authors propose a novel strategy to jointly optimize a Neural LIDAR Field and the corresponding LIDAR poses. They achieve this by utilizing a hybrid strategy of alternating pure graph-based geometric optimization and global optimization of the Neural Field. Additionally they introduce a new selective reweighting strategy that steers gradients in the Neural Field towards outlier poses and new geometric losses that improve geometric reconstruction. They perform experiments on two datasets and achieve better results than relevant baselines in both reconstruction and registration.

Strengths

1. The proposed method with its hybrid optimisation and other components is novel and very interesting and achieves impressive results relevant to the field of LIDAR simulation. 2. The paper is well written and gives deep insights into the optimisation procedure with well designed figures (e.g. Figure 3&4). The concepts are very clearly laid out and supported by experimental evidence and ablations. Some sentences could be improved but this is very minor and does not detract from the overall paper. 3. It is very hard to find technical faults in this paper. The experiments are comprehensive and consider relevant baselines and the introduced components are backed up with mathematical foundations quite well.

Weaknesses

In order of severity: 1. Is it truly Pose-Free? The starting point for the LIDAR pose optimization is not completely random but they are perturbed from Ground truth by noise with a standard deviation of 20deg in rotation and 3m in translation (line 235). This means that some form of reasonable initialization is necessary for the method, which would have to be obtained from somewhere. How to obtain this initialization is not discussed in the paper. 2. The motivation for why this approach is necessary is unclear. The setting of obtaining sparse LIDAR scans without at least poses derived from GPS or vehicle odometry seems unrealistic, as well as the assumptions that cameras would not be available given that they are significantly cheaper than LIDAR Scanners. This subtracts from the otherwise strong paper because the setting and significance of the results cannot be properly contextualized. It is also not clear from the Introduction what motivation the simulation of LIDAR data in a Neural Field has and what potential applications are enabled by the proposed method. 3. While the authors report standard pose registration metrics (RPE & ATE) they do not consider Inlier Ratio, Feature Matching Recall or Registration Recall as metrics. This makes a comparison to prior work such as GeoTransformer [45] more complicated since these are the main metrics which they evaluate on in their paper. ([45] Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., & Xu, K. (2022). Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11143-11152)) 4. Outlier/Inlier poses and frames are discussed in Figure 3 and for example line 208 but it is never clarified how their inlier/outlier status is determined. This makes reproducibility harder and should be clarified either in the paper or supplementary. 5. In section 3.5 the authors introduce a new loss between generated and ground truth point clouds. It is not mentioned how the generated point cloud is obtained from the Neural Field, which is important to know since this determines how the gradient from the loss is backpropagated through this mechanism. This should also be clarified in the final version or supplementary. 6. In Equation 13 the variable P is very confusing since in the equations above it represents a point cloud and then here probably refers to the ray-drop loss. This is in contrast to line 119 & 120 where intensity and ray-drop probability are introduced as $S$ and $R$ respectively. This should be made coherent with each other or at least clarified. 7. In line 113-115 the authors explain how the direction of rays is determined but do not explain how they represent translation in their polar coordinate system. This could also be clarified in the supplementary. Minor comments: - Line 16 NeRFs has achieved → NeRFs have achieved - Line 49 In furtherance of mitigating overfitting → To reduce overfitting. Clarity > Complicated Words. - Line 301 demonstrate the promising performance → demonstrate promising performance.

Questions

To focus discussion about the weaknesses raised above here are some detailed questions: 1. How can the initial (bad) registration of the point clouds be obtained or is there a way this step can be removed ? Can the point cloud poses be initialized completely randomly or all as identity? 2. Is there some reason (e.g. Privacy concerns) that prohibit the joint capture of Image data via cameras and LIDAR scans? This would enable standard Structure from Motion approaches for pose estimation. 3. What is the overall motivation of representing LIDAR in a Neural Field and what benefits are gained by this? Are there downstream applications enabled by the proposed method? 4. Can you report the Inlier Ratio, Feature Matching Recall or Registration Recall ? If not, why can this not be reported on this data?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitations of their work with regards to dynamic scenes, since they only consider static scenes. Societal impact is not discussed.

Summary

The paper proposes GeoNLF, a method for pose-free Lidar point cloud novel view synthesis. They propose optimizing point cloud poses simultaneously through a bundle-adjusting neural lidar field as well as through ICP-inspired geometric optimization. The neural lidar field is supervised with ground truth range images, intensity images, and ray-drop probability images as well as ground truth point clouds. The geometric optimization optimizes poses to minimize a weighted average of the chamfer distances between neighboring lidar frames. The proposed method is validated for lidar novel view synthesis and pose estimation using the Nuscenes and KITTI-360 datasets.

Strengths

1) The authors apply knowledge from nerf pose estimation to the problem of lidar novel view synthesis through the bundle-adjusting lidar nerf. 2) They propose a novel alternating pose optimization between the implicit nerf representation and the explicit point clouds. 3) The proposed method is supported by experiments on two real-world datasets which verify its validity. 4) The paper proposes a new task of jointly recovering pose and lidar point cloud novel views, which is significant for lowering the impact of noisy estimated lidar poses. 5) They achieve state-of-the-art results for lidar novel view synthesis with the proposed method, as well as better point cloud registration.

Weaknesses

The methodology section reads like a work in progress, making it hard to understand the details of the proposed method. 1) There is no intuitive explanation as to why the "J" term in Eq. 5 can simply be dropped. While the main text states that dropping this term allows independent updates of rotation and translation, the supplementary material then states that simultaneous optimization yielded satisfactory results in L527. 2) P is used to denote poses in Eq. 1 as well as point clouds in Eq. 11 and ray-drop loss probabilities in Eq. 13. L173. Similarly, I is used to refer to NeRF images in L112 and then intensity images in Eq. 13. Intensities and ray-drop probabilities are referred to as S and R in L119. 3) L173 states that Eq. 7 shows how CD is in line with each step of ICP, but the optimization steps of ICP are never introduced. 4) Eq. 8 introduces d_clipped which is never used. It takes as parameter a distance "d" that is never introduced. 5) There is no explanation as to how the normals used in Eq. 12 are obtained. Are they obtained from the implicit density field for both predicted and pseudo-groundtruth pointclouds? 6) There is a lot of information in Fig. 3, but the caption does not explain what the figure is trying to show. 7) Sections 3.2 and 3.5 both discuss the nerf optimization, but are separated by sections 3.3 and 3.4 which discuss geometric optimization.

Questions

Please refer to the weakness section.

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

3

Limitations

There is no discussion about the optimization time, which might be significantly increased due to the geometric optimization requiring computing the chamfer distance between all combinations of the 4 adjacent frames.

Comments on Rebuttal

Dear Authors, thank you for the comprehensive rebuttal. I read it and my questions were addressed well. I would strongly encourage the authors to include the additional explanations provided here for Q2, Q3 in the introduction section of the final version of the paper, since this motivation was (in my opinion) missing in the initial submission. The additional experiments and clarifications provided in the rebuttal would also be nice to see in the final paper. I will raise my score to Accept.