One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection

Summary

This paper proposes OneDet3D, a universal one-for-all model that addresses 3D detection across various domains, including both indoor and outdoor scenes. It tackles two primary issues: data-level interference caused by differences in the point clouds themselves and category-level interference caused by label conflicts among categories across different domains.

Strengths

1. The paper introduces a universal one-for-all 3D detection model applicable across different domains, using only one set of parameters. 2. The proposed method demonstrates superior performance and remarkable generalization ability compared to existing state-of-the-art approaches.

Weaknesses

1. In lines 60-62, the authors claim that 3D sparse convolution is better than point-based feature extractors due to its robustness to domain gaps. Could the authors elaborate on this in detail? 2. Regarding the domain router, how is the domain label n obtained? Does data from the same dataset share the same domain label? 3. In Table 1, different datasets have different views. I wonder if any design could be implemented to tackle this difference for better generalization ability. 4. In Equation 1, experiments regarding the hyperparameters α and c are missing. The values of these hyperparameters are not mentioned. 5. In Table 2, the best performance achieved by existing methods is not highlighted in bold. 6. Table 4 underlines the second-best performance, while Table 5 does not. Table 4 underlines the second-best performance, while Table 5 does not. It would be beneficial to maintain consistency.

Questions

Please refer to the weakness section.

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

N/A

Summary

This paper proposes OneDet3D, which is a multi-domain jointly trained point cloud object detector for universal 3D object detection. It is the first 3D detector that supports point clouds from both indoor and outdoor scenes simultaneously with only one set of parameters. The experiments are conducted on multiple indoor and outdoor benchmarks.

Strengths

- The proposed techniques are simple yet effective with intuitive explanations. - This work showcases that multi-domain training can achieve better performance than single-domain training, which is a valuable observation.

Weaknesses

- The performance in Table 2 for the nuScenes dataset is strange, where AP is not aligned with the numbers reported by other methods. - The sizes of the adopted datasets are quite different. Are any resampling strategies adopted? How to mix them up? The training scheme and implementation details should be added, such as training schedule, data augmentation, and so on. - The attributes of point clouds are different. For example, indoor datasets may contain RGB information and outdoor data such as Waymo may contain elongation and timestamp information. How to deal with the different channel sizes? A lot of details are ignored by the authors. - There are many basic grammar issues. For example, "is joint training on" should be "is jointly trained on". - I notice the authors claim and adopt fully sparse architecture. However, there is no comparison or discussion with previous work with fully sparse structures such as [1 - 4]. What are the differences between the proposed architecture and theirs? Do the issues they addressed still exist? - Sec. 3.3 is hard to follow and should be rewritten with more details and sufficient explanation. (1) The description of "we utilize language vocabulary embeddings from CLIP" should be more detailed. Does the network predict CLIP embedding? How to supervise? (2) Why is it necessary to convert sparse point features to dense features in Sec. 3.3? (3) What is the specific definition of the dense features? (4) Why is the back propagation unfeasible? (5) How many classification branches does it have? What does the "both branches" in L215 mean? - Lack of comparison with other similar methods such as [5,6] in Table 4 & 5. \ [1] Fully Sparse 3d object detection \ [2] Voxelnext: Fully sparse voxelnet for 3d object detection and tracking \ [3] FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxels \ [4] SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection \ [5] ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection \ [6] Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection

Questions

See the weakness part. I would like to increase the score if my concerns are well addressed.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors do not discuss the limitations and social impacts, which should be added.

Summary

This manuscript introduces OneDet3D, a universal point cloud-based 3D object detector designed to address the challenges of multi-domain joint training. The primary motivation is to overcome the limitations of existing 3D detectors, which are typically trained and tested on single datasets, restricting their generalization across different domains. OneDet3D aims to provide a unified solution that can handle diverse indoor and outdoor scenes using a single set of parameters. The authors claimed the following contributions of OneDet3D: - Domain-Aware Partitioning: This technique aims to address data-level interference caused by differences in point cloud structures across domains. The parameters related to data scatter and context learning are partitioned and guided by a domain router, allowing the model to learn domain-specific features without increasing complexity. - Language-Guided Classification: By incorporating text modality through CLIP embeddings, OneDet3D mitigates category-level interference among different datasets. This approach allows for better generalization to new domains and categories. - Fully Sparse Architecture: The use of 3D sparse convolution and an anchor-free detection head makes the model robust to domain gaps and efficient in handling point clouds from various domains. Experiments on datasets like SUN RGB-D, ScanNet, KITTI, and nuScenes demonstrate the effectiveness of OneDet3D. The model achieves promising performance in both closed-vocabulary and open-vocabulary settings, showing improvements over existing methods and strong generalization abilities.

Strengths

(+) OneDet3D addresses the challenge in 3D object detection by introducing multi-domain joint training, which enhances the model's ability to generalize across various indoor and outdoor scenes. Such an endeavor is in line with the current research trend for point cloud 3D perception. (+) The manuscript includes extensive experiments and evaluations on multiple datasets, showcasing the model's superior performance and generalization capabilities compared to some state-of-the-art methods. (+) The overall OneDet3D framework seems standard and scalable; with more datasets and larger model sizes involved, the performance could be further improved.

Weaknesses

(-) The overall OneDet3D framework is a combination of several previous approaches, which might not demonstrate a strong novelty in the related area: - The domain router and context partitioning from Sec. 3.2 is similar to what [R1] and [R2] did for reducing domain differences. - The scatter partitioning method in Sec. 3.2 is closely related to Uni3D [R3] (see their Sec. 3.3). - The language-guided classification in Sec. 3.3 shares the same intuition with [R4] and [R2]. (-) The experimental comparisons could benefit from involving more recent 3D object detectors. For example, the latest model in Tab. 2 (UVTR) is from two years ago; while most of the other models are from 2020 (or even earlier). (-) The overall elaboration of this manuscript deserves further improvements. Several claims are made without supporting evidence or references. Besides, most of the technical details are given in the text; having more graphical illustrations or algorithm flows could reduce the redundancy in the elaboration and help readers better understand the proposed method. --- ### References: - [R1] Towards Universal Object Detection by Domain Attention. CVPR, 2019. - [R2] Multi-Space Alignments Towards Universal LiDAR Segmentation. CVPR, 2024. - [R3] Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection. CVPR, 2023. - [R4] DaTaSeg: Taming A Universal Multi-Dataset Multi-Task Segmentation Model. NeurIPS, 2023.

Questions

- **Q1:** As mentioned in Weakness 1, it is recommended to make a clearer comparison and have additional discussions with closely related works from existing literature. Putting the Related Work section behind the Introduction section and adding more detailed analyses and discussions with existing works can be beneficial. Including experimental comparisons and ablation studies with existing works, such as Uni3D, could further justify the effectiveness of the proposed OneDet3D. - **Q2:** As mentioned in Weakness 2, supplementing the results with more recent single-dataset training approaches could further improve the comprehensiveness of the benchmark studies. - **Q3:** As mentioned in Weakness 3, the manuscript could benefit from having more graphical illustrations or algorithm flows, instead of just plain text descriptions. This is also in line with the style of ML conferences. - **Q4:** For the scatter partitioning method in Sec. 3.2: Since the dataset-specific statistics are used, how does OneDet3D handle a new point cloud with unknown statistics during inference? - **Q5:** How does OneDet3D handle the class mapping differences using the language-guided classification in Sec. 3.3? For example, how to handle cases like the different definitions of `bicycle` and `bicyclist` (in KITTI and Waymo)?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors acknowledge several limitations in their approach: The current focus on supervised learning with fully annotated point clouds limits scalability. Future work should explore weakly labeled or unlabeled data to reduce reliance on extensive annotations. Additionally, the manuscript lacks a detailed analysis of the computational cost associated with the proposed methods, which is crucial for assessing real-time application feasibility. What is more, there is a risk of overfitting to seen domains during multi-domain training. More experiments on unseen domains would strengthen the claims of generalization. The scalability and generalizability of OneDet3D to other types of sensors and different environmental settings have not been extensively discussed. Further research is needed to assess the model's robustness in diverse real-world scenarios.