Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection

Summary

The authors explore and analyze the existing vehicle-infrastructure cooperative 3D object detection framework, and propose to adopt flow-based rather than still images to extract temporal features from multiple LiDAR input frames. The experimental results on DAIR-V2X dataset is better compared to the existing methods in terms of car objects.

Strengths

1. The task of vehicle-infrastructure cooperative 3D object detection is very important in the 3D community and has real-world practical applications. Interestingly, the authors propose to extract temporal features for different input frames. Also, they propose a self-supervised method to extract the flow-based features.

Weaknesses

1. The evaluation of one single class on a single dataset is not strong enough to prove the effectiveness of the proposed method. It would be convincing if the authors could conduct the experiments on other labeled classes, i.e., bus, truck, or van. Also, experiments on another existing dataset would be better as well. 2. Memory footprint comparison. The authors didn't make a comparison of their work in terms of memory with the existing vehicle-infrastructure cooperative 3D detection methods. As far as I am concerned, the memory consumption/additional efforts are needed since the compression and transmission are included.

Questions

Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

This work proposes a flow-based feature fusion framework called Feature Flow Net (FFNet) for vehicle-infrastructure cooperative 3D object detection (VIC3D). FFNet can generate aligned features for data fusion to transmit information for fusion and solve uncertain temporal asynchrony and transmission costs. The proposed self-supervised training of the feature flow generator lead FFNet to mitigate temporal fusion errors across various latencies. The results on the DAIR-V2X dataset show superior performance compared to other cooperative methods while consuming only about 1/100 of the transmission cost of raw data and using a single model.

Strengths

- The writing is clear and easy to follow. - Using predicted features to solve the latency issue is neat and effective. - The whole pipeline is simple but well-designed. It covers the transmission speed and bandwidth while keeping the model performance in the meanwhile. - The results are solid and expressive. For different latency scenarios, FFNet achieves the best performance with less transmission data.

Weaknesses

- The abbreviation of models is not easy to distinguish. For instance, FFNet versus FFNet-V2 versus FFNet-O, etc. - The ablation of compression is missed. Without proposed compression and decompression, will it affect the tolerance of the latency of transmission? - The last paragraph of the ablation study is not clear. The definition of the test part of the Dataset is unclear. If the models are exposed to the dataset test set, then it will be better than the one that doesn't (FFNet-O) for sure.

Questions

- Although the latency and the bandwidth problem is indeed a real-world challenge, the Vehicle-Infrastructure Cooperative looks a little unrealistic. In that case, the autonomous driving vehicle will only be able to use such kind of extra information when there is an infrastructure sensor. Wouldn't it?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No.

Summary

This paper introduces FFNet, a flow-based feature fusion framework that incorporates a feature flow prediction module to predict future features and addresses asynchrony issues. Instead of transmitting feature maps extracted from static images, FFNet transmits feature flow by leveraging the temporal coherence of sequential infrastructure frames. To evaluate the effectiveness of FFNet, experiments are conducted on the DAIR-V2X dataset.

Strengths

1. The motivation is clear and the idea is simple yet effective. 2. The overall results look good.

Weaknesses

1. The major limitation of this work pertains to its narrow scope, focusing exclusively on vehicle-infrastructure cooperative 3D object detection and mainly addressing the latency issue. While the proposed idea demonstrates effectiveness within this specific application, its applicability to broader contexts (such as multiple vehicle scenarios) remains uncertain. 2. Another weakness lies in the experimental section. The comparison made with V2VNet and DiscoNet seems unfair since these models were not originally designed to address communication latency. To ensure a more comprehensive evaluation, it would be beneficial for the authors to incorporate additional modules into these methods or compare their approach to more advanced models, such as SyncNet (ECCV 2022). This would provide a more accurate assessment of the proposed method's performance in comparison to state-of-the-art techniques. 3. A minor weakness is that the literature review is not sufficient. The authors should include more closely-related works such as [1-5] [1] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R. and Yu, H., 2023. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13712-13722). [2] Li, Y., Zhang, J., Ma, D., Wang, Y. and Feng, C., 2023, March. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning (pp. 2062-2072). PMLR. [3] Xu, R., Tu, Z., Xiang, H., Shao, W., Zhou, B. and Ma, J., 2023, March. CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers. In Conference on Robot Learning (pp. 989-1000). PMLR. [4] Li, J., Xu, R., Liu, X., Ma, J., Chi, Z., Ma, J. and Yu, H., 2023. Learning for vehicle-to-vehicle cooperative perception under lossy communication. IEEE Transactions on Intelligent Vehicles. [5] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023. Uncertainty quantification of collaborative detection for self-driving. ICRA.

Questions

Considering the major concern about the limited scope and application, the authors may consider submitting this work to a more appropriate venue.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N.A.

Summary

In this paper, a cooperative detection framework, named Feature Flow Net (FFNet), is presented to address temporal asynchrony and limited communication condition challenges in vehicle-infrastructure cooperative. Specifically, FFNet transmits feature flow to generate aligned features for data fusion, providing a unified manner to transmit valuable information for fusion while addressing the challenges of uncertain temporal asynchrony and transmission cost. Incorporating with self-supervised learning, the proposed FFNet framework presents a performacne improvement for VIC3D object detection with less communication cost.

Strengths

1. FFNet transmits feature flow to generate aligned features for data fusion. This idea is simple yet can transmit temporal asynchrony information for fusion with less transmission cost. 2. The proposed self-supervised training approach enables FFNet with feature prediction ability to mitigate temporal fusion errors across various latencies without cooperative view and labeling. 3. FFNet demonstrates superior performance compared to other cooperative methods and demonstrates robustness in terms of latency.

Weaknesses

1. For 3D object detection, map@3D proves the detector accuracy better than map@BEV. However, it can be seen from Table 1 that FFNet's performance advantage in map@3D is much smaller than that of map@BEV, and the advantage on IoU@0.7 is less than IoU@0,5. This may indicate a) The proposed method does not learn the changes in the height of the environment, resulting in poor accuracy in high dimensions of FFNet. b) The flow-based prediction strategy is not accurate enough, which limits the upper quality of FFNet detection results.

Questions

1. Experiments and visualizations are expected to analyze and verify why FFNet is deficient in high-quality detection boxes. 2. Can a finer supervision method, not limited to the global similarity, improve the performance of FFNet on map@3D?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See Weakness.

Authors have addressed most of my concerns.

Thanks for the answers and clarification in the rebuttal, which covered most of my concerns.