CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection

Summary

This paper proposes a radar-camera and temporal fusion method (CRT) for the 3D object detection task. CRT design Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF) modules. MVF employs radar information for 2D-to-3D projection. MGTF utilizes velocity and occupancy predictions from MFE to fuse multi-frame BEV features. The proposed method achieves state-of-the-art 3D object detection results on nuScenes.

Strengths

1. The MGTF module is interesting in decoupling and fusing multi-frame object features on dense BEV features. 2. The paper achieves a good balance between performance and computational efficiency, making it a promising advancement for real-time radar processing in autonomous driving. 3. The paper achieves a SOTA for radar-camera 3D object detection

Weaknesses

1. The proposed method requires velocity supervision, while some datasets, such as Waymo, do not provide velocity labels. 2. Table 1 shows that the proposed has well speed-accuracy trade-offs. It is better to provide the inference time of each part in the model. 3. It seems that hyperparameters have a significant impact on model accuracy (Table 9). 4. The paper does not show the model generalization of the proposed method. It is only applied to BEVDepth. 5. BEV segmentation in MFE is misleading. It is better to use terms like object occupancy or foreground segmentation.

Questions

1. Training details. How is the model trained? Is the multi-frame multi-modal model trained in one step? 2. What about the training cost? 3. What TTA is used in Table 2? 4. In Line 153, a set of M radar features are associated with W. What M does the model use? What if radar features are less than M? 5. What is B_{t−k}(x, y) in eq.4？Please give more details for object occupancy ground truth generation. 6. What do O and X mean in Table 5?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitations.

Summary

This paper presents a method, called CRT-Fusion, for 3D object detection that fuses temporal information with radar-camera features. The CRT-Fusion captures the object motion with three modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module uses Radar-Camera Azimuth Attention (RCA) to enhance the depth information in camera BEV features and combines them with radar features through a gated fusion network. The MFE module estimates pixel-wise velocity and performs BEV segmentation, which guides the temporal feature alignment and fusion in the MGTF module across multiple timestamps. The evaluation on the nuScenes dataset indicates that CRT-Fusion achieves state-of-the-art performance.

Strengths

This paper is well-written and easy to follow. While there is prior work on fusing temporal and camera-radar features to improve 3D detection, the presented method appears novel as it learns how to estimate object pixel-wise velocity and occupancy to guide the fusion of object-level temporal features. The proposed Radar-Camera Azimuth Attention (RCA) is shown to effectively enhance image features with depth information from radar features. The experiments are well-designed, and the results on the nuScenes dataset show a significant performance gain over SOTA.

Weaknesses

The paper could improve clarity in the following: 1. As mentioned in lines 181-185, CRT-Fusion uses object-level pixel-wise velocity and occupancy to integrate temporal information with BEV features. CRT-Fusion's performance relies on the accuracy of both detection and velocity estimation. It is not clear how to address error propagation across frames. 2. Regarding inference latency, some approaches (e.g., CRN[6]) use only 2 adjacent frames as input. According to the results in Table 8 of the supplementary material, the 2-frame version of CRT-Fusion performs similarly to (or worse than) CRN in terms of NDS and mAP metrics. This raises concerns that the performance gains of CRT-Fusion may be a trade-off involving longer temporal cues, leading to increased latency and complexity. The paper should discuss this trade-off. 3. The batch size used during inference is not specified; Table 7 only presents the batch size used during training. Additionally, it appears that the authors directly used speed results from other papers for comparison, which may not be meaningful as the machines used for different models can vary significantly (Unfortunately, some other papers also directly cite and compare the speed) .

Questions

In line 212, should it be $M_{t-k}(x, y)$? In the inference, the model does not have access to ground truth.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper poorly discusses limitations. Failure cases and potential improvements of the approach in the future should be discussed.

Summary

In this study, the authors present CRT-Fusion, an innovative framework designed to incorporate temporal information into radar-camera fusion, thereby enhancing the robustness of 3D object detection. The CRT-Fusion framework is composed of three integral modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF).

Strengths

This paper presents a novel framework capable of integrating temporal information into radar-camera fusion for enhanced 3D object detection. Overall, the paper is well-written and clearly articulated. Experimental results demonstrate that this method outperforms previous approaches in 3D object detection.

Weaknesses

Please consult the question section for further information.

Questions

1. The paper lacks a detailed description of the loss function used in the proposed framework. 2. The authors claim that the Multi-View Fusion (MVF) module enhances depth prediction. However, no experimental evidence is provided to substantiate this assertion. 3. The Motion Feature Estimator (MFE) module is responsible for the pixel-wise velocity estimation task, yet the paper does not clarify how the velocity estimation is supervised during the learning process. 4. The overall improvement achieved by the proposed method is limited. Given that the method incorporates temporal information, velocity estimation, and depth estimation tasks, its performance enhancement compared to the CRN framework is relatively modest.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please consult the question section for further information.

Summary

The paper proposes CRT-Fusion that fuses radar and camera inputs for 3D object detection in BEV space. A radar-camera azimuth attention module is proposed for improving image features for better motion awareness, which further improves the quality of image BEV features. A motion-guided temporal fusion is introduced to align dynamic objects in history BEV feature maps with their positions in the current BEV feature map, using predicted object presence information and predicted velocities. The authors conduct experiments on the nuScenes dataset and prove the effectiveness of the proposed model.

Strengths

1. The proposed framework is concise and straightforward, while showing good performance on nuScenes benchmark 2. The proposed RCA first decouples the image feature map to row and column features, and only applies cross-attention on column features, which enhances image features with a relatively low computation cost. 3. The motion guided temporal fusion mitigates the BEV feature misalignment issue of moving objects across different timestamps, and improves the 3D object detection performance.

Weaknesses

1. The proposed temporal fusion requires aggregating multiple feature maps of different timestamps in a recurrent way. The inference latency and required GPU memory will grow linearly with the number of past frames. 2. The multi-view fusion section needs more details. It's unclear how the radar BEV feature map is obtained and how the gated fusion is performed. It's better to include some equations for illustration. 3. The equation 7 for motion-guided temporal fusion seems to be incorrect. Further details for generating the shited feature map are needed. For example, let's consider this situation: the velocity predictions at (x1, y1) and (x2, y2) are (Δx1, Δy1) and (Δx2, Δy2) respectively, and x1+Δx1 = x2+Δx2, y1+Δy1=y2+Δy2, what will the shifted feature map be at (x1+Δx1, y1+Δy1)?

Questions

In Table 1, I notice that CRT-Fusion-light has lower mATE, mASE, mAOE, mAVE, mAAE compared to CRT-Fusion, which is unexpected. It would be beneficial if the authors could provide a detailed explanation or revisit the experimental setup to clarify these results.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

The paper talks about multi sensor fusion using Motion Information for Bird’s Eye View Object Detection. Authors have presented a multiple step approach to improve the object detection on nuScenes dataset. The proposed approach comprises of three parts; Multi-View Fusion (MVF) that enhances depth prediction by leveraging radar features to improve image features before fusing them into a single BEV vector; a Motion Feature Estimator (MFE) step that estimates pixel-wise velocity information and Motion Guided Temporal Fusion (MGTF) step that iteratively aligns and fuses feature maps across multiple timestamps. The proposed approach is tested on the latest nuScenes dataset for object detection and obtains state-of-the-art results on the dataset surpassing many publicly listed approaches.

Strengths

1. The paper is easy to read and follow. Authors have described each step carefully and mentioned proper mathematical formulation for each step. 2. The motivation and improvements in the performance is clear. In many cases the proposed framework archives state of the art results. 3. Ablation study is also very well supported. Authors have included results for baseline modules and how each of their component improves the performance of object detection.

Weaknesses

1. I believe that the technical novelty of the paper is somewhat limited. The idea of MVF fusion, i.e. Radar and Camera image feature fusion has been studied well in the past. 2. The paper lacks qualitative results completely. No where in the main paper have the authors presented any qualitative analysis. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails. 3. Overall approach with multiple components, although produces state-of-the art results, but is nearly impractical to scale or even productionize in a real world setting. Authors haven’t mentioned anywhere in the paper about this limitation (only described it briefly in the Supplementary section). The limitations of the current approach should be more broadly presented in the main paper, i.e. if the number of frames increase, the model will have high computational cost and hence not scalable.

Questions

1. Can authors provide a small description of how this model can be made a bit efficient such that this can be deployed in the real world? 2. Figure 2 is not very clear. For the MVF (bottom left side) figure, what does the middle gray box represent? 3. Line 135 states an LSS approach, but that hasn’t been mentioned anywhere. What is LSS? 4. Line 137 states “existing state-of-the-art…”, can authors please cite which work they are referring to here? 5. The calculation of GT for MFE is not very clear. Why do we use the input for MVF to compute the GT for MFE (line 199)? 6. For line 201, how did the authors determine the value for T_{iou}? Why did they consider 0.5? Was there some ablation study conducted? 7. For Equation 7, it’s not very clear what B^_{t-k} is? Also how was the value for T_{v} chosen here? 8. Table 1 here refers to TTA. What is TTA? 9. Can the authors provide some qualitative results? And compare the existing work with the proposed approach to show what are the strengths of their approach? Just listing down metrics may not be motivating enough to see where the proposed approach does best. 10. Line 283 states that CRT does not do well in Sunny conditions which is a bit strange since sunny weather conditions should be ideal for object detection. Can the authors provide some explanation here? 11. For table 6, RCA has worse performance as compared to RGVT on mATE. Can authors address why is that the case?

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Authors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.

The rebuttal successfully addresses my review comments

The rebuttal successfully addresses my review comments. The rebuttal presents additional results that address my concerns about the latency of the proposed method. I would like to keep my original rating.

Response to rebuttal

Thank you authors for presenting the rebuttal documents. I have read through the comments and the attached PDF. Since most of my concerns are addressed, I am okay in changing my rating from "borderline reject" to "borderline accept". There are still some concerns on the complexity of the solution but given that most of the issues have been addressed, I am inclined towards borderline acceptance here.