Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection

Summary

This paper introduces a novel cross-modal BEV distillation approach, namely VCD, but adopts identical feature extractors for the teacher and the student model, which eases the distillation difficulties during the knowledge-transferring phase. (1) It proposes a new multi-modal teacher VCD-E, with an image-based backbone only to serve as the teacher model, and achieves similar performance with multi-modal 3D detectors. (2) It introduces a novel trajectory-based feature distillation approach, to enhance the feature distillation quality on moving objects. (3) The final model achieves 63.1 NDS on nuScenes test leaderboard, setting SOTA performance for camera-based detectors.

Strengths

1. The paper points out that a LiDAR-based detector may not be essential in camera-based 3D detectors' distillation: a dedicated image-based detector with future frames temporal fusion and LiDAR-guided depth input is enough. Based on this, the author overcomes the domain gap between LiDAR and images easily. 2. The paper presents a novel trajectory-based distillation approach to deal with moving objects, which is simple yet effective according to ablations.

Weaknesses

1. The paper lacks some theoretical analysis, which is practical work. 2. Though the trajectory-based distillation is effective, the reviewer does not think VCD truly captures the moving objects. A more detailed experiment may help.

Questions

1. The official BEVDet4D-depth is 51.9NDS, while the author report 54 NDS. What's the modification to the model? 2. The reviewer is interested in the static objects' performance similar to Figure 1(supplementary). It is not straightforward to see moving objects' improvements only, since VCD has already improved over its baseline. 2. The author is suggested to clarify that VCD-E utilizes future frames during training. 3. The author is suggested to use \times rather than x in Table 1, which is not formal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This work presents vision-centric distillation, which utilizes multi-modalities for the expert and images for the student. It includes two distillation parts, namely Trajectory-based Distillation and Occupancy Reconstruction. Experiments prove the effectiveness of the proposed method.

Strengths

1. The proposed method tries to improve the performance via distillation from depth (Occupancy) and motion (Trajectory), which is reasonable for image-based 3D detection. 2. Experiments show significant improvement over various benchmarks. 3. The paper presentation is overall clear.

Weaknesses

1. It is interesting to see the work improves the framework from depth and motion. However, the definition of Occupancy in Figure 2 and Section 3 could be confusing. Because there is no annotation for occupancy optimization in the framework, which cannot guarantee occupancy generation. It's more like depth supervision from expert to apprentice from my point of view. 2. Because the motion trajectory is generated from the predicted velocity. The difference between Trajectory-based Distillation and previous prediction-based distillation should be made clear. The classic prediction-based distillation also includes velocity for teacher-student optimization. 3. What's the advantage of a.2 over a.1 in Figure 1(a). Because this work focuses on the camera-based setting. And the backbone for experts (not included in inference) utilizing the image-only or modality-specific approach in Figure 1(a) seems not to make an inherent difference. The essential advantage of the point cloud is accurate depth, which is already utilized in this framework. 4. It's better to add the results StreamPETR[37] in Table 2, which achieves better results. 5. It's unclear how the LiDAR-camera depth fusion in Figure 2 is conducted.

Questions

Please refer to the Weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Broader Impact is given in the supplementary material.

Summary

This paper aims to design a multi-modal expert teacher with little domain gap to distillate the LSS-based 3D object detector. Different from existential work, it just leverages the LiDAR depth information to design a teacher model instead of using a cumbersome LiDAR feature extractor. The proposed method shows effectiveness to transfer the knowledge from teacher to student detector, which shows significantly results.

Strengths

1. Without the aid of a LiDAR feature extractor, it designs an apprentice-friendly multi-modal expert to conduct distillation. 2. VCD-E achieving comparable performance with state-of-the-art multi-modal methods who predominately rely on the LiDAR backbone.

Weaknesses

1. The comparison with other multi-modal methods should report the latency to highlight the superiority. 2. The performance seems to be saturated since the performance of VCD-A with ConvNeXt-B and 8 frames is not good. The Baseline is the BEVDepth with 2 frames. If the frame is set to 8 frames, what is your improvement compared with the baseline?

Questions

When using a large backbone like ConvNeXt, you can not provide any evidence to verify the proposed method can boost performance.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see the weakness section.

Summary

The paper presents an innovative approach for improving camera-only 3D object detection. It introduces a vision-centric multi-modal expert and a trajectory-based distillation module to address key challenges in the field. The framework includes an apprentice-friendly multi-modal expert and a fine-grained trajectory-based distillation module to rectify motion misalignment for each object in the scene. The proposed VCD-A model achieves state-of-the-art performance on nuScenes dataset.

Strengths

1. The paper presents an innovative approach of incorporating a vision-centric multi-modal expert that exclusively relies on camera features, eliminating the need for a LiDAR backbone. This approach simplifies the model architecture while delivering comparable performance to multi-modal methods. 2. The trajectory-based distillation module alleviates the problem of motion misalignment in long-term temporal fusion, improving the accuracy of object detection. 3. This paper provides clear figures and tables, making it easy for readers to understand and follow.

Weaknesses

1. Typesetting problem, Reference[23] and Reference[24] are repeated. 2. The Fine-grained Trajectory-based Distillation Module dose not really solve the problem of dynamic target detection, but only enhances the feature representation through multi-frame alignment, thereby improving performance.

Questions

1. Could you provide provide the dynamic object detection performance gain brought by the trajectory-based distillation module? 2. You can add a column about the number of frames used in the table. As far as I know, the number of frames used by different methods is inconsistent. How many frames does the VCD use?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

How to improve the detection performance of dynamic objects is an urgent problem to be solved at the current stage of 3D perception. The starting point of this article is promising. I hope that the authors can conduct more in-depth research on the real improvement of dynamic object detection performance.

Summary

This paper seeks to improve camera-only 3D object detection by distilling a multi-modal expert model. The authors start by developing a multi-modal expert with essentially the same architecture as the camera-only model - intended to reduce the domain gap. Then, they propose trajectory-based distillation as well as occupancy reconstruction to supervise the student. The resulting camera-only model achieves state-of-the-art in nuScenes, and the multi-modal expert is a simple, strong method in its own right.

Strengths

- The intuition that the expert should have similar architecture to the student is intuitive. - The proposed expert model’s significant increased performance when using a stronger image backbone in Table 7 is interesting. - Extensive experiments demonstrate that having an aligned architecture between the teacher and student is helpful (BEVDepth teacher outperforms TransFusion as well, which is a nontrivial observation). - The final model achieves state-of-the-art in nuScenes.

Weaknesses

- The Occupancy Reconstruction section is difficult to understand. It appears that the predicted expert depths are outprojected to XYZ. However, it is unclear how the 3D voxel grid for occupancy reconstruction is generated. Further, it’s actually not clear how the outprojected XYZ is used, as G_xyz is defined in (5) and used in (6) without further mention of the outprojected XYZ. In L207, Should G_xy be G_xyz? It is also unclear how this optimizes depth prediction capabilities for static and dynamic objects. I would appreciate a clear explanation of this part, as it is one of the main contributions of this work. - What model/settings was used for the Table 4 ablation? The numbers seem too drastically worse compared to Table 1. In Table 1 in the supplementary, the baseline VCD-A model in the first and second sections appear to be the same, but the results differ. - While Trajectory-based Distillation improves performance in Table 9, additional ablations are missing. More specifically, the “simplest” form of Trajectory-based distillation would be to simply just distill GT instance locations in the current frame, without considering past trajectory locations. This ablation appears to be missing.

Questions

- Additional details on the expert model are necessary. How many past frames were used to accumulate LiDAR to create the sparse depth map? Does using past LiDAR not cause significant depth artefacts for moving objects?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

2 fair

Contribution

4 excellent

Limitations

The authors have included a limitations section.

Response to Author

The rebuttal addresses most of my concerns. Therefore, I keep my rating. ps. it would be better to state VCD-E adopts future frames for training in your main context, rather than in the supplementary.

Response to authors

Thanks for the rebuttal from the authors. It addresses most of my concerns. So, I'd like to improve the rating to Borderline Accept.