Unleash the Potential of Image Branch for Cross-modal 3D Object Detection

Summary

This paper proposes a novel cross-model 3D detector BiProDet, which leverages the information from image domain in two ways. First, it proposes point-to-pixel bidirectional propagation strategy to boost the representation ability of the point cloud backbone. Second, it introduces NLC map estimation as an auxiliary task for training to learn local spatial-aware features of image branch to supply sparse point clouds. The BiProDet exhibits consistent and more significant improvements on the "moderate" and "hard" levels, where objects are distant or highly occluded with sparse points and ranked 1st on the KITTI 3D detection benchmark for the cyclist class. In my opinion, this paper discusses of the reinforcement mechanism of image branch for 3D detector deeply, and a bidirectional propagation is designed to enhance the performance of the backbone, which is very innovative. And a wealth of ablation experiments has been done to prove the effectiveness of the algorithm proposed in this paper.

Strengths

The paper has several strengths in terms of originality, quality, clarity, and significance. 1. It introduces a 2D auxiliary task called normalized local coordinate (NLC) map estimation to improve the performance of the cross-modal detector by providing the relative position of each pixel inside the object. 2. The paper proposes a novel point-to-pixel feature propagation mechanism that allows 3D geometric features from LiDAR point clouds to enhance the representation ability of 2D image learning. 3. It provides a analysis of the performance bottlenecks of single-modal detectors.

Weaknesses

1. The motivation is weak. As the proposed method point-to-pixel is just a fusion strategy in cross-modal modeling, which also been explored in many works such as BEVFusion[1]. So it is relatively weak to serve as the motivation of the paper. 2. The proposed specific 2D auxiliary tasks have been explored in TiG-BEV [2]. Althogh the format of the 2D tasks is not the same, the core of the methods is similar. [1] https://github.com/mit-han-lab/bevfusion [2] TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning

Questions

None

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

None

Summary

This paper studies 3D object detection with multi-modal inputs (image and point cloud). This study uses a two-stage 3D object detection pipeline and proposes two approaches for further performance improvements. Firstly, the authors propose a bidirectional feature module to fuse point cloud and image features. Secondly, the authors propose several 2D and 3D auxiliary tasks to improve representation learning. The experiments validated the proposed approaches on KITTI and Waymo Open Dataset.

Strengths

1. the ablation study is well-designed and extensive to validate the effectiveness of the proposed NLC map representation and fusion network. 2. the idea of using NLC map is well-motivated. 3. the performance on KITTI and Waymo Open Dataset is competitive in several metrics.

Weaknesses

1. the technical novelty is somewhat limited. Bidirectional feature fusion is not new, for example, see "FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation". The proposed fusion approach is similar to this paper, which is not cited. In addition, using auxiliary loss to improve representation learning is also not very novel. 2. Auxiliary tasks require extra labels. Do other approaches in Table 1 use semantic masks as supervision? 3. Compared with [45], the performance in the EASY category is inferior. 4. Previous fusion-base approaches such as [1] provided evaluations on the Nuscenes dataset yet this paper does not.

Questions

1. Why the performance on the EASY category of KITTI is inferior compared with [45]? Can the authors further improve it? 2. Can the authors provide quantitative comparisons with [1] on the Nuscenes dataset? 3. have the authors attempted other fusion approaches instead of the proposed one? What will be the performance difference?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have not addressed the limitations.

Summary

This paper propose a multi-modal usion-based 3d object detector named BiProDet. BiProDet adopts a bidirectional feature propagation mechanism, i.e., point-to-pixel module and pixel-to-point module. Besides, BiProDet propose a new auxiliary task called Normalized Local Coordinate (NLC) map.

Strengths

The paper is well written, and the ablation study demonstrate the effectiveness of the proposed method.

Weaknesses

1. Comparison with some closely related work is missing. For example, the proposed method is similar to EPNet++ ( [21] in the submitted manuscript) in several aspects. Both of them design bidirectional feature fusion and adopt semantic segmentation as an auxiliary task. 2. BEV representation is receiving increasing attention recently. However, a comparison with these methods is missing.

Questions

1. The proposed method is closely related to EPNet++. It would be better to provide a more detailed comparison between these two approaches. 2. BEV representation is receiving increasing attention recently and achieves sota results on many benchmarks. It would be better to compare the proposed method with BEV-base fusion methods (e.g. BEVFusion). 3. It would be more convincing to evaluate the proposed method on more datasets besides KITTI, e.g., Nuscenes. 3. Why NLC is more beneficial for small objects such as ped. and cyc.?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The proposed fusion method is not as flexible as the BEV representation. And the comparison with BEV-based methods is missing.

Summary

This work addresses the task of 3D object detection from LiDAR and cameras. Their main contribution is developing a joint 2D and 3D stream architecture, with simple bidirectional feature flow in the backbones. To improve this, they propose to predict NLC maps in the image stream. The proposed components demonstrate performance improvement on the KITTI dataset.

Strengths

- The paper is clear and easy to read, with diagrams to explain crucial parts. - The NLC prediction significantly improves performance, supporting the core hypothesis in the paper. - As opposed to some existing works with joint 2D-3D backbones, the proposed bidirectional feature flow is simple without bells and whistles and complicated components. - It is a non-trivial and interesting result that point-to-pixel flow during training only can still improve 3D detection performance, showing that 2D gradients can help 3D model learning.

Weaknesses

- The statement “However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.” in L41 - L46 (as well as its encompassing paragraph) is unclear to me and seems unsubstantiated. Many existing works (PointPainting, AutoAlign, etc) have enhanced the point cloud/3D voxels with 2D semantics before/while using the 3D backbone network and have demonstrated significant improvements in detection metrics by training end-to-end. - I am slightly concerned that all of the ablative results shown are on the KITTI dataset, which is a small-scale dataset with low diversity. I would appreciate some ablations on at least the key components (NLC prediction, point-to-pixel flow) on the Waymo dataset. Nonetheless, it is impressive that a point-based method can achieve such strong performance on Waymo. - Point-to-pixel flow during training only does improve 3D detection performance, which seems to demonstrate that training a 2D CNN can help the 3D network learn better features. However, this is slightly confounded with the possibility that *NLC* supervision is what is improving 3D detection when point-to-pixel flow is added, not the 2D CNN. It would strengthen the paper if point-to-pixel flow improves performance even when 3D NLC supervision (similar to that in PartA2) is done.

Questions

- Please reference the weaknesses section for questions. - I also want to ask if any augmentations are done in 2D. For instance, LR flipping may introduce ambiguity in predicting left vs right side of an object.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Authors do not appear to have included a limitations section.

Summary

This paper proposes a method for multimodal (image and LiDAR) 3D object detection. The main purpose of the proposed method is to enhance the image branch. The authors design the task of NLC Map estimation, which is to predict the normalized local coordinates of points within a ground-truth box. The prediction happens on the image plane and the points are projected (carrying their NLC labels). In addition, image semantic segmentation is also adopted as an auxiliary task for the image branch. The image and LiDAR branch fuse their features during the forward of their backbones. The LiDAR branch predicts 3D instances based on the RPN proposals and RoI-pooled fused features.

Strengths

1. Although the idea of predicting local coordinates within a bounding box is similar to Part-A^2, it is interesting to see that it can effectively enhance the image branch. I am also wondering if additionally introducing local coordinate prediction for the LiDAR branch can further boost performance. 2. The proposed method achieves SOTA results on the KITTI dataset, and the effectiveness of the auxiliary tasks are proved by ablation study.

Weaknesses

1. The experiments are only conducted on the KITTI dataset, in fact, there are a bunch of recent multimodal 3D detection methods that provide results on the NuScenes dataset, so it would be more convincing with comparisons on NuScenes. 2. The bidirectional feature propagation module is simply gathering features from another modality by coordinate projection and feature transformation, which is not significantly novel.

Questions

Would it be more straightforward to use local coordinate prediction in the LiDAR branch like Part-A^2? NLC estimation can also be performed for the points if they are not projected to the image plane?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors did not address the limitations.

Looking forward to hearing from you

Dear Reviewers, Thank you for taking the time to review our submission and providing us with constructive comments. We would like to inquire if our responses have adequately addressed your earlier concerns. Additionally, if you have any further concerns or suggestions, we would be more than happy to address and discuss them to enhance the quality of the paper. We eagerly await your response and look forward to hearing from you. Best regards, The authors