Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion

Summary

This paper presents a method to perform image and raw radar fusion for object detection from a Mobile platform. The authors explain that the commonly used radar data processing pipelines discard too much data, resulting in very sparse point clouds. In contrast, the proposed method operates on raw data and performs end-to-end filtering within the model with intermodality cross-attention. They consider the RADIal radar dataset, for which they provide more precise and exhaustive annotations to better evaluate their method's added precision.

Strengths

- The motivation and results are convincing, and the analysis interesting - The model is natural and exploits the unique properties and structure of radar acquisitions - Extensive ablation study and new annotations - Not a lot of work on this interesting and promising modality exists

Weaknesses

- The method section could be more precise and detailed. The CrossAttn operator is not defined with enough details (what are its arguments, Why are arguments 2 and 3 are always the same?), and neither are polar queries. There is a reference to [8] at the end of the method which helps a little bit, but the paper should be readable on its own. As it stands, the cross-attention method is not reimplemetable. - While submissions about sensors not typically treated at NeurIPS are welcome, the community is not necessarily familiar with the subtlety of radar data. As it stands, the submission overly relies on jargon and complex facts presented as self-evident. The figures and their caption could be more helpful. The authors should be more didactic. - The writing could be significantly improved, as it contains many imprecise and clumsy formulations.

Questions

This submission presents a well-motivated and designed method with promising results. However, the presentation makes it hard to follow the methodology, and the lack of pedagogy makes it hard to understand for non-radar experts. However, the weaknesses should be mostly fixable. Q1) What are the arguments of CrossAttn? Why are arguments 2 and 3 are always the same? Q2) What are polar queries? Are they different from object queries? Q3) "The polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension" This sentence is hard to parse. What does this mean for a coordinate system to be aligned with a type of data? Q4) Figure 1 is unclear and hard to parse. Which one is the proposed method? Whar exactly are the methods compared? What are the red circles supposed to highlight? S1) Figure 2 could be great for explaining concretely the nature of the data presented. However, it is full of jargon and acronym and not clear at all. Maybe on a simpler example? Or even a toy example?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

2 fair

Contribution

4 excellent

Limitations

good

Summary

This paper proposes a Polar-Aligned Attention module to guide the fusion of radar data and images. The module projects the image and radar features onto a polar coordinate system to calculate the similarity of the associated information. This effectively enhances the direct fusion of radar data and images. Additionally, the authors have restandardized the RADIal dataset, improving its accuracy and standardization.

Strengths

The authors claim that this paper is the first work to directly fuse radar data with image data, achieving the fusion of relevant features through cleverly designed attention mechanism modules. Additionally, the authors' reannotation of the dataset has played a positive role in the development of this research direction.

Weaknesses

Although this paper proposes a polar-coordinate-based attention mechanism module for the fusion of radar and image data, the contribution of such a structure is limited. Furthermore, due to the lack of dataset support for the research task of radar and image fusion in this study, the experiments were only conducted on an existing dataset with modifications. Therefore, I do not have enough confidence to affirm the sufficiency of the research motivation in this paper.

Questions

1. How is the alignment of different modalities ensured when fusing the features of image and radar data? 2. Is it possible to visualize the alignment relationship between the two modalities in the polar-coordinate attention mechanism?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Due to the limitations of the dataset used in this study, it is not possible to extensively and sufficiently demonstrate the motivation behind the research in this paper.

Summary

This work presents a method for camera-radar detection and fusion. The authors propose an approach that follows: Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023. and Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022. Specifically, the paper introduces a polar transformer method that cross-attends image and radar features across modalities according to view and sampling geometry. The technique relies on RAW radar measurements as input and, similar to recent works (above), is evaluated on an existing dataset. The approach is validated to be on par with recent methods.

Strengths

The cross-modal fusion is well-motivated and has shown to be effective. Although a large body of work has proposed similar methods that also rely on RAW radar data, the design of the proposed method is sound and well-executed. Devising methods that operate on RAW data is an exciting opportunity for the field to embrace ideas from computational imaging in vision models.

Weaknesses

The proposed method lacks novelty and experimental margins. Specifically, areas of concern are the following: *Novelty and Technical Contribution: A large body of work has explored BEV-based fusion approaches and, recently, RAW-based detections. Specifically, this work directly follows the core ideas from: Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023. and Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022. While the specific attention mechanism is new, it also extends the formulation in PolarFormer to the radar measurement formation model. As such, the technical contribution of the method is incremental in comparison to existing work in the field. *Experimental Margins: As a results of the relatively incremental changes in architecture and training, the margins that the proposed method achieves are also also incremental. The method is on par with FFTRadNet on the original dataset, but improves after labeling. However, it is unclear if the baseline method has been retrained and all other baseline methods are also missing in the dataset. Moreover, without validating the relabeled dataset separately, it is unclear if the relabling biases the evaluation.

Questions

Can the authors include examples of mislabeled objects or bias in the dataset? What scenarios are corrected and why do they contribute significantly to the evaluation?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discuss the limitations of their work and societal impacts of the method adequately.

Summary

The paper proposes a method for fusing raw radar data with images in bird’s eye view (BEV) space. To this end, the method extract multilevel features from images and radar and uses polar-aligned attention mechanism with polar-BEV decoder for bounding box prediction. Evaluation is done on re-annotated RADIal dataset and shows better performance than other comparable methods.

Strengths

The paper utilizes raw radar data which in general is a better approach than relying on post-processed radar data for object detection.

Weaknesses

The BEV cross attention assumes synchronized radar and image which may not be robust at high speed driving and unsynchronized radar and camera setup.

Questions

How robust is the proposed model to spatio-temporal misalignment?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The method assumes synchronized radar and camera.

Summary

The paper proposes a new method to fuse radar and imagery data for autonomous driving. Instead of using processed radar data, which discards most of the information in the original signal while attempting to filter out noise, this method leverages the raw data directly as input to the network. The method is based on attentional mechanisms cued to range in radar and azimuth in imagery, as these measurements are more accurate in those modalities. This technique and the use of raw data seem quite novel in this domain. The experiments are very thorough, on the only dataset that provides raw radar data. Full 3D bounding box annotations are added to this dataset, which is a useful contribution.

Strengths

The introduction clearly lays out the motivation of using raw vs. processed radar data, and Fig. 1 provides a clear example of how this improves performance. Lines 44-46 state “This technique is based on an observation that the polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension.” It’s not clear with “aligned with” means here, but it seems you mean to say that radar data is more informative in the range dimension while imagery is more informative in the azimuth direction. It would be helpful to clarify this important point in the paper. Adding vehicle bounding boxes to RADial is a significant contribution for those looking to study radar in autonomous driving. It would be useful to expand the annotations beyond a single “vehicle” class however, as with most of the other autonomous driving datasets. The related work section is thorough and covers the main areas of claimed contributions. The overall architecture, Fg. 3, nicely highlights the key contribution of the cross-attentional module in polar coordinates that emphasizes azimuthal accuracy in imagery and range accuracy in radar. The approach in sec. 4 is clearly explained in text, supported by equations but without relying on those equations to encode the concepts, making the paper more readable and impactful. The experiments are thorough, albeit on a single dataset as mentioned below. The primary comparative method [36], which operates on the processed radar data, performs considerably worse on the most important test, table 3, against the newly-created 3D bounding box annotations. The proposed method also does very well against all baselines using the standard point annotations on the original dataset. The ablation studies are very thorough, examining performance on each sensor independently in order to show the value of fusion.

Weaknesses

Experiments are performed on a single dataset, which is a minor limitation but unavoidable because it is the only one that provides raw radar data calibrated with imagery. The focus of the paper is autonomous driving, which is a huge topic and is generally sufficient for papers at major AI conferences. Nevertheless it would be compelling to see how the method performs in other domains where radar and imagery could be fused, such as pedestrian detection and tracking, maritime vessel detection and so on. Datasets for these are hard to find, but perhaps the RADial dataset contains pedestrians that might have some motion signature in the radar data.

Questions

As above.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, no concerns.

Follow up

I appreciate the comprehensive responses from the authors and the additional experiments, which provide valuable insights. The authors have effectively addressed many of the concerns that could be addressed as a rebuttal. However, a significant concern persists: the paper's readability. Some challenges arise from the writing, which could benefit from further refinement in termsof rigour, clarity, and didacticity. However, the primary obstacle is the inherent complexity of the RADAR modality. I encourage the authors to approach the presentation in a more pedagogical manner, without assuming the NeurIPS readership's familiarity with this specific modality. Given the considerable potential of RADAR and the contribution this paper brings to the table, I'm inclined towards recommending its acceptance, with the understanding that the authors will continue working on the paper's readability. Broadening the scope of accepted papers to include less modalities beyond text and image, like RADAR is the only way to incrrase the community's awareness, understanding and interest in these complex but powerful sensors. Wether the paper is accepted or resubmitted at another CV/ML conference, the authors should prioritize: (i) Enhancing the writing quality to ensure it doesn't add to the subject's inherent complexity. (ii) Articulating concepts as clearly as possible, prioritizing clarity without compromising precision. Deeper technical details can be relegated to the appendix for interested readers and radar experts. Regarding Q2, the uncommon nature of this approach warrants explicit mention to avoid any ambiguity.