Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

Summary

This paper introduces a framework for registering 2D images and 3D point clouds. The framework consists of three branches, for processing 2D images, 3D point clouds and 3D voxel constructed from the original 3D point clouds. The fused features of 3D point clouds and 3D voxel are used to establish correspondences with the features of the 2D images, and a differentiable PnP solver is used to calculate the transformation for end-to-end training. Experiments are conducted on two public datasets, KITTI and nuScenes, and the proposed method outperforms several existing baseline methods.

Strengths

1.The three-branch structure is novel for this task and seems effective in improving the registration performance. 2.The proposed method achieved much higher performance than existing methods in the experimental setting. 3.The paper is generally well written and easy to follow.

Weaknesses

Major 1.My major concern is the practical meaning of the proposed work. I wonder if there is a real scenario where this kind of registration is essential. Why not establish the correspondences between 2D images and 3D point clouds by device calibration? If we calibrate a LiDAR and an image camera, which should not be difficult, we can register the data captured with it (including both images and point clouds) to either an image or a point cloud by mono-modal registration. This two kinds of mono-modal registrations are both well studied with much higher accuracy. 2.The ablation study has some issues. First, I notice that ablation is conducted on a different setting from the main experiment. Why not use the same setting as in Table 1? In the ablation study, only sequence 0-1 is used for training and only sequence 7 is used for testing, which are much less than the main experiment and makes the ablation results unstable. Second, from the ablation study we can see that removing any one component results in a light decrease of performance, but all ablation study performance are still much higher than baseline methods in Table 1. Existing ablation results can not show where the performance gain of the proposed method comes from. Further analysis is needed, such as removing two or three components at the same. 3.The data are split into training and testing sets, but the author did not mention if a validation set is used and how they chose the best model for inference. 4.RANSAC is used for more robust registration of the proposed method. We know that RANSAC plays a very important role in correspondence-based registration. Can RANSAC also be used in the comparing methods to improve their performance? Minor 1.The references [31,32] and [19,22] in the first sentence of the second paragraph of Introduction are not proper, since these papers does not study the problem of establishing 2D-3D correspondences. 2.In the introduction of the KITTI datasets, “2D translation on the ground within ±10”, what’s the unit of the translation? 3.“registration accuracy (Acc.)” is not a good name for its current definition. Registration recall may be better. 4.The inference time is 0.19s, which can not be said as real-time.

Questions

Please respond to questions raised in weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

limitation is not discussed.

Summary

This work aims to address the image-to-point cloud registration task. The authors propose a VoxelPoint-to-Pixel matching framework, which consists of three network branches dedicated to extracting features from voxel, point, and pixel representations, respectively, for 2D-3D matching. The network is trained with four different losses: an overlap prediction loss, a circle loss for 2D-3D feature matching, a KL divergence loss for probabilistic PnP, and a pose loss supervised by ground-truth. To assess the effectiveness of their approach, the authors perform experiments on two existing LiDAR datasets, showing improved 2D-3D matching performance compared to existing methods.

Strengths

- The proposed 2D-3D matching framework combines several well-established techniques. To bridge the domain gap between points and pixels, the authors suggest employing a triplet network to extract element-wise 2D and 3D features. To enhance the robustness of the 2D-3D matching process, the authors incorporate an overlap prediction branch. The whole pipeline is supervised by a 2D-3D matching (circle) loss and pose estimation (PnP) losses. - The authors assess the matching performance on the KITTI and nuScenes benchmarks, demonstrating that their method has better performance when compared with DeepI2P and CorrI2P. - The paper is generally easy to comprehend. I appreciate the presence of suitable illustrations accompanying the explanations in Sec. 3.

Weaknesses

- One main concern for this work is its limited technical innovations: the adaptive-weighted loss is basically the circle loss [39], while the KL divergence loss for probabilistic PnP is borrowed from Epro-PnP [8]. Additionally, there is a lack of ablation studies for those four losses - it is unclear how the KL divergence loss $L\_{KL}$ and the pose loss $L\_{pose}$ contribute to the matching performance and whether both of them are necessary. - For the experiments on the KITTI and nuScenes datasets, the comparisons are limited to DeepI2P and CorrI2P. However, several other works related to 2D-3D matching are absent from the comparisons, such as the ones below. Additionally, it would be interesting to assess the generalization of the proposed method across datasets, for instance, from KITTI to nuScenes. - Hierarchical Scene Coordinate Classification and Regression for Visual Localization. CVPR 2020. - P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching. ICCV 2021. - For the ablation study in Sec. 4.6, it is unclear why only sequences 0-1 are used. It would be more beneficial to include all available data in order to thoroughly validate the design choices. - Typos: - L267, “mis-registration”? - L309 “dis-matched”?

Questions

Please see the Weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors did not discuss the limitations of their method. It would benefit the reader to include a failure case analysis, which would provide a more comprehensive understanding of the proposed approach.

Summary

The paper proposes a 2D to 3D registration pipeline using a differentiable PnP method (Epro-PnP) and integrates 3D information using both voxel and point-based features. These designs target on previous problems such as the domain differences when fusing MLP-based point features and CNN-based pixel features, and the non-differentiable post-processing when computing the transformation that cannot train the entire model end-to-end. Experiments on KITTI and nuScenes, along with ablation studies show the effect of the proposed pipeline while achieving great efficiency.

Strengths

- The proposed method improves the current state-of-the-art methods by a large margin in terms of the accuracy and efficiency on KITTI and nuScenes datasets. - Although most techniques used in the paper are not new, combining these techniques yield effective results. - The idea of combining both the voxel-based features and the point cloud-based features helps with the domain difference between MLP-based and CNN-based features. - The writing of the paper is okay and is easy to follow.

Weaknesses

Major: - In the ablation study Table 3, the authors showed that the differentiable PnP solver improves the accuracy of the method. Could the authors further explain the reasons? - In Line 287-288, the authors mentioned that they did not exclude extreme cases when reporting the performance. However, the authors could report both the performance with/without extreme cases for a clear and fair comparison to other methods. - The paper uses non-differentiable PnP to solve pose during inference. What about other methods applying a fast PnP during inference? Will they achieve similar speedups? - Ablation study (Table 3): how about using 1 or 2 of the components to show a more comprehensive study of the effectiveness of the combination of the module? - Table 4 better adds efficiency comparison. - The limitations of the method should be discussed in the main paper. Minor: - The attached code does not show many intuitions. A pseudo code is preferred in the text to aid the understanding of the method. - There are some typos in the paper that need to attend to.

Questions

Please address my questions/concerns above. Specifically, my main concerns are some unclear/inconsistent arguments and experimental designs.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation of the method is not discussed. Some of the limitations that can be discussed are as follows: 1. The method uses a non-differentiable fast PnP solver during the inference, which is different from the optimization process. How will it affect the performance? And will any fast PnP solver be suitable here? What is the limitation during inference? 2. What is the limitation of the method when applied to different datasets/scenes? 3. How to choose model parameters in order to balance accuracy and efficiency? The authors have included some ablation studies in both the main paper and the supplementary materials. However, the tradeoff between accuracy and efficiency could be further discussed.

Summary

This paper proposes a method to register an image with its nearby Lidar scan, and it comes with three modules: 1) A sparse 3D conv-net and point-net to extract 3D features; A 2D conv-net to extract 2D features; 2) An intersection detection module to discard non-matchable 2D and 3D points; 3) Modified Circle, Probabilistic PnP, and pose losses are used to train the proposed triple network.

Strengths

1) This paper is well-written and easy to follow; 2) This paper makes good practice to combine multiple existing modules to address the image2Lidar registration problem; 3) According to Table 1, the proposed method outperforms previous works addressing the image2Lidar registration problem.

Weaknesses

Though this paper successfully combines multiple existing modules, some design choices are yet to be investigated. Specifically, 1) Generalization ability. According to the paper, the proposed method focuses on autonomous driving datasets and conducts experiments on 3-DoF registration problem (Line 267). I have a concern that the network overfits on this specific configuration. To address my concern, please conduct the following experiments: a) Cross-validation. Using a network trained on the KITTI dataset to test on the nuScenes dataset; b) Conducting 6-DoF registration, rather than 3-DoF registration. Please at least add rotations around the x-axis and y-axis; 2) Pipelines. a) The effectiveness of the modified Circle Loss. Please add the comparison with respect to the original Circle Loss in Table 3; b) Since metric-learning plays an important role in this paper, I would expect authors to conduct comparisons with off-the-shelf metric-learning losses other than the Circle Loss. Please refer to https://github.com/KevinMusgrave/pytorch-metric-learning; c) The effectiveness of the Probabilistic PnP and pose loss. I would expect a two-stage training strategy here by first training the network by only using the adaptive-weighted loss, and then adding the Probabilistic PnP and pose loss. This reminds me of the work [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization]. Please separate the impact of KL divergence loss and pose loss, and check their effectiveness independently in Table 3. It would be nice to further compare the proposed Probabilistic PnP with respect to the differentiable PnP module in [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization]. 3) Minor. I think processing 2D and 3D data with convolutions may not justify the claim of "Sharing similar characteristics in feature space" (Line 162), as 3D convolution is invariant to shift, viewpoint, distance, etc. In contrast, 2D convolution is vulnerable to viewpoint and scale differences.

Questions

Please refer to the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The paper proposes to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver, which designs a triplet network to learn VoxelPoint-to-Pixel matching. The proposed method is trained in end-to-end manner by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. The experiments seem good.

Strengths

(1). A framework to learn Image-to-Point Cloud registration by learning a structured cross-modality latent space with adaptive-weighted optimization, together with an end-to-end training schema driven by a differentiable PnP solver. (2) The paper represents the 3D elements as the combination of voxels and points to overcome the pattern gap between points and pixels, where a triplet network is designed to learn VoxelPoint-to-Pixel matching.

Weaknesses

1. The motivation that use voxel information is unclear, as described in the paper, one of the bottleneck of previous methods is points and pixels are with different characteristics with patterns learned in different manners (MLP and CNN), however, domain gap is also exited between voxeled information and pixels, why does the paper use voxel information should be discussed. 2. The ablation studies are not enough, (1). in 3.2 Adaptive-Weighted Optimization, a hyper-parameter radius r is used, which is should be discussed, because it affects the positive and negative pairs, experiments should be provided to verify the robustness. (2). In probabilistic PnP, the effectiveness of L_{pose} and L_{kl} should be verified separately. (3). I agree that the settings in CorrI2P is unsuitable that exclude RTE larger than 5m and RRE larger than 10◦, however, it is better to provide these results same with CorrI2P, since it would be better for readers to understand where does main improvement come from. (4). Limitatations and failure cases should be discussed.

Questions

Please see weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations and failure cases should be discussed.

Response to all: Thank you very much for the thorough reviews.

We deeply appreciate the reviewers for their thoughtful feedback and time invested in evaluating our work. We are delighted that reviewers appreciated the representation and the significance of the paper. We are heartened by Reviewer Ehvn's commendation on the novelty of our method and by both Reviewer CqXY and Reviewer ec8L's acknowledgement on the capability of our work to "bridge the domain gap between points and pixels". We are also pleased that the Reviewer yFFc found our method "improves the current state-of-the-art methods by a large margin". We are excited by Reviewer QyPU‘s characterization of our work as "makes good practice". Moreover, we are encouraged with the recognition of reviewers that our paper is "well-written", "easy to follow" and "with suitable illustrations accompanying the explanations". We make a global response with a rebuttal PDF containing some experimental results and pseudo codes requested by the reviewers. We further respond to each reviewer separately. Thank you again for your invaluable feedback and we are looking forward to continuing the discussion.

comments on response

My concerns on this papere are well addressed in the response though some of them can only be included in further revision because of the time limit. Especially, I appriciate the detailed ablation study conducted in the response. I update my score to weak accept.

Comments on Authors' Rebuttal

I appreciate the authors' effort in addressing most of my concerns, including technical contributions and loss ablations. One addtional comment, regarding the claim in the response, "almost no work explored the cross-modality contrastive learning between image features and voxel-point features", there is one seemingly related work I am aware of, though it is on transfer learning: - Liu et al. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. 2021. Another question about the loss ablation study: were these new quantative results obtained on all available data of KITTI or still sequences 0-1?

Response to the authors

Thank the authors for providing additional information and experiments. I have read all the comments of the authors and the reviewers, and I appreciate that the authors could further add the ablation studies and the limitation discussions. Most of my concerns were addressed. I did not agree with the authors' response about the generalizability of the proposed method. However, the additional Table C may provide an example of the generalization of the method. The limitations I have listed were just some examples. I hope the authors could discuss this in the main text. In conclusion, I encourage the authors to improve the paper based on all the comments from the reviewers.