MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders

Summary

This paper proposes a monocular 3D detection framework inspired by Masked Autoencoders (MAE), designed to address the challenge of object occlusions in 3D object detection. It utilizes a unique depth-aware masking module that simulates occlusions by adaptively masking non-occluded object features based on depth information, coupled with a lightweight completion network that reconstructs these masked features to learn occlusion-tolerant representations. It generates training pairs of non-occluded and occluded object representations directly, enhancing its capability to handle occlusions effectively. The framework is optimized for low computational overhead during inference, as it does not require object masking at this stage.

Strengths

1. The proposed method outperforms the conventional methods across various datasets such as KITTI and Nuscenes. It demonstrates the effectiveness of the proposed method. Moreover, the proposed method achieves real-time inference time. 2. An extensive ablation study is proven to demonstrate the impact of the proposed module. 3. The idea is simple yet effective.

Weaknesses

1. The performance improvement is marginal, especially on the cross-validation in Table 6. 2. Missing evaluation on the Waymo dataset

Questions

1. Many recent 3D object detection studies have utilized the Waymo dataset for evaluation. Could you explain why your experiments were limited to KITTI and nuScenes? 2. There appears to be a performance drop in the nuScenes dataset at distances beyond 40 meters. Could you provide insights into what causes this decline? 3. There is a slight difference in inference time between 'Ours*' (36ms) and 'Ours' (38ms), with significant performance differences noted in Table 2. Could you elaborate on the role of the completion network (CN) given these differences? 4. The mask ratio r varies with scale parameters and maximum depth. How sensitive is your method to changes in the mask ratio?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

See the weakness and question parts.

Summary

This paper introduces a novel framework for improving monocular 3D object detection, particularly in handling object occlusions. The proposed MonoMAE leverages depth-aware masking to simulate occlusions in the feature space and employs a lightweight completion network to reconstruct occluded object regions, thereby learning occlusion-tolerant representations. Experiments show that this learning stratgy helps to improve the performance of monocular 3D object detection.

Strengths

1. This paper is well-structured, with a clear problem statement, methodology, experiments, and ablation studies that substantiate the contributions and effectiveness of MonoMAE. 2. This paper addresses a significant challenge in monocular 3D object detection, object occlusion, with a novel approach using depth-aware masked autoencoders.

Weaknesses

1. The reliance on depth-aware masking to simulate occlusions may not perfectly replicate natural occlusion patterns, potentially affecting the model's reconstruction accuracy. The gap between synthetically masked and naturally occluded object queries could limit the model's robustness in real-world scenarios. 2. While this paper claims generalizability, the lack of extensive cross-dataset validation leaves the true scope of its generalization capability somewhat unproven.

Questions

1. All the experimental results presented in this paper are about vehicle detection. Does MonoMAE also work for more difficult cases like pedestrain and cyclist detection? 2. The paper suggests investigating generative approaches for simulating natural occlusion patterns. Can you elaborate on what this might entail and how it could further improve monocular 3D detection?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

1. The paper could provide a more detailed analysis of the computational efficiency, including speed and resource usage, to fully assess the practicality of MonoMAE for real-time applications. 2. This paper only present results of vehicle detection. The performance of detecting objects with small sizes is unknown.

Summary

This paper applies Masked Autoencoder to 3D object detection. It distinguishes object queries into occluded and non-occluded categories, and during training, it applies depth-aware masking to the non-occluded queries and learns by completing them. At test time, the completion is applied to the occluded queries.

Strengths

- It achieved state-of-the-art performance on the KITTI 3D dataset. - The idea of interpreting occluded queries as masked queries to solve the problem is interesting. - The training and test times are illustrated clearly in figures.

Weaknesses

- As stated in the limitations section, occlusion at the image level and masking at the feature level of object queries are not the same. Further analysis is needed to understand the actual implications of masking in object queries. - If masking serves the role of occlusion at the image level, there should be no reason for the mask ratio to vary with depth, yet depth-aware masking is highly beneficial. An analysis is needed to understand why depth-aware masking works well compared to random masking. - In my opinion, the performance of the Non-Occluded Query Grouping classification is crucial for the framework to function properly. Although classification accuracy is provided in the supplementary material, it would be helpful to include various metrics such as precision, recall, and F1-score. If the results of the Non-Occluded Query Grouping classification are biased, it might be interesting to apply completion not only to the occluded queries but also to the non-occluded queries at test time.

Questions

Please refer to the weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are included in the main text.

Summary

This paper introduces MonoMAE, a novel monocular 3D object detection framework designed to improve detection performance in the presence of object occlusions. MonoMAE leverages the concept of Masked Autoencoders, treating object occlusions as natural masking and training the network to complete occluded regions. This innovative approach addresses the pervasive issue of object occlusions in monocular 3D detection, leading to superior detection performance. Extensive experiments on datasets like KITTI 3D and nuScenes show that MonoMAE outperforms state-of-the-art methods in both qualitative and quantitative measures.

Strengths

1. The introduction of depth-aware masking to simulate occlusions and the use of a lightweight query completion network are innovative and address a significant challenge in monocular 3D detection. 2. MonoMAE improves detection performance without the need for additional training data or annotations, making it a practical solution for real-world applications like autonomous driving and robotics. 3. The framework demonstrates superior performance on benchmark datasets (KITTI 3D and nuScenes), outperforming existing state-of-the-art methods in both occluded and non-occluded scenarios. 4. MonoMAE shows strong generalization capabilities to new domains, which is critical for deploying models in diverse environments.

Weaknesses

1. In many datasets and methods, objects are not merely labeled as "occluded" or "non-occluded." Instead, they may be assigned occlusion levels or degrees that quantify the extent to which an object is occluded. These levels provide more granularity and can influence how models are trained and evaluated. It would be beneficial to specify how occlusion levels are defined and used. Clarifying whether discrete or continuous levels are employed and how these influence the labeling, training, and evaluation processes will provide a clearer understanding of the methodology and its robustness in handling occlusions. 2. The paper does not provide explicit details about the accuracy of the occlusion classification network or how this accuracy influences the overall 3D object detection network. This information appears to be missing. 3. The paper does not explicitly report the performance or accuracy of the query completion network. Including a report on the performance of this network, such as quantitative results or visualization of the reconstructed queries, would be valuable. It would demonstrate whether the query completion network is learning meaningful features and contributing effectively to the overall 3D object detection performance.

Questions

1. What is the accuracy of the occlusion classification network? How does the accuracy influence the whole 3D object detection network? 2. What is the accuracy of the query completion network?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed some failure cases in their paper, as well as the gap between the generated occlusion pattern and natural occlusion patterns.

Provide some qualitative results

Thanks for providing additional information in your rebuttal. Your reply has addressed part of my concerns. Regarding the part "Question 1 & Limitation 2: Performance on Small Size Objects (Pedestrian and Cyclist)", providing more qualitative or visualization results would be more convincing.