UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes

Summary

This work presents an unsupervised 3D object detection method named UNION, which exploits LiDAR, camera, and temporal information jointly for generating pseudo bounding boxes to train existing object detectors. In addition, the authors introduce an appearance-based clustering method to generate pseudo class labels and train the object detector in a multi-class fashion.

Strengths

++ The method has significant improvement on the accuracy compared to existing unsupervised methods. ++ This method does not require the time-consuming multi-round self-training procedure.

Weaknesses

-- There are not any qualitative results in the paper. Visualizations on the pseudo box/class generation and the comparative results would help readers understand the method more comprehensively. -- Detailed analysis of failure cases would make this paper stronger. For example, in which scenarios UNION would generate bad pseudo labels, and how each step of UNION contributes to different failure cases. -- Details about the HDBSCAN hyper-parameters are missing.

Questions

-- Could the authors explain why OYSTER and LISO are not included in the multi-class experiments? -- Will self-training help improve the detector trained with UNION pseudo labels? -- For visual appearance encoding, DINOv2 usually can only get a lower-resolution feature map of the input image. How did the authors obtain point-wise features? -- Which variant of DINOv2 is used for the experiments in this paper? -- Could the authors provide the runtime analysis for each step of UNION?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are discussed in the paper.

Summary

This paper explores the challenge of unsupervised 3D object detection, introducing UNION. UNION leverages camera, LiDAR, and temporal information jointly to train 3D object detectors without relying on self-training. The approach demonstrates strong performance particularly on the nuScenes dataset. Additionally, the authors tackle unsupervised multi-class 3D object detection by clustering object appearance embeddings and employing these clusters as pseudo-class labels.

Strengths

1. This paper studies a significant problem in unsupervised 3D object discovery. 2. The paper is clearly written, and its motivation is straightforward. 3. This method introduces visual embedding to enable certain static objects to be generated during object proposal generation. 4. This method has achieved good performance on nuScenes dataset.

Weaknesses

1. Compared to prior work, the method's novelty is weak. The core innovation lies in leveraging self-supervised visual features to aid in object proposal extraction, thereby eliminating the need for self-training to acquire static vehicle data. 2. The core exploration of this work revolves around object appearance embedding. The current study only investigates the features of DINOv2. It remains to be explored how other self-supervised features perform in comparison, or whether combining different self-supervised methods could yield better results. 3. As for training the multi-class detector, I believe the choice of K is crucial. The authors use K=10 as default. It should conduct an ablation study on the impact of K. 4. As depicted in Table 2, the discussion centers around the challenges in orientation estimation. To enhance clarity, it would be beneficial to include detailed metrics such as ATE, AOE, etc. 5. In spatial clustering, when utilizing point cloud aggregation, how to deal with the dynamic objects, such as multiple moving cars whose point clouds may overlap. 6. As shown in Table 4, the AP for cyclists is always zero. Does this imply limitations of the method for detecting long-tail objects? What are some potential directions for improving detection of these less frequent objects in the future?

Questions

See weakness.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations of the work.

Summary

The paper introduces UNION, an unsupervised 3D object detection method designed to detect both static and dynamic objects without manual labels. UNION utilizes spatial clustering and self-supervised scene flow to generate object proposals and employs visual appearance encoding to distinguish between static and dynamic objects. This approach enables a single training process, enhancing performance while reducing computational costs. Experiments on the nuScenes dataset show that UNION significantly improves the state-of-the-art performance in unsupervised object discovery, doubling the average precision score to 33.9.

Strengths

1. The approach is quite novel, successfully eliminating the dependence on self-training. 2. The presentation is clear and easy to understand. 3. The paper effectively highlights the similarities and differences with related literature.

Weaknesses

The experiments are relatively limited. 1. Besides DINOv2, how do other image encoders affect the model's performance? 2. In Table 4, the cyclist AP is 0. Is there a more detailed investigation into the reasons for this? Given that, as a small object, the pedestrian class shows a significant improvement in AP with UNION-10pc compared to UNION+Size prior, why does the cyclist class not exhibit a similar improvement? 3. In Fig. 3, 5% was chosen as the threshold. Why was this value selected? The authors should provide more discussion or experimental support for this choice.

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors acknowledge the limitations of their work and plan to address these in future research.

Summary

This paper introduces UNION, a novel method for unsupervised 3D object detection that leverages object appearance-based pseudo-classes. This paper addresses the challenge of training object detection models without manual annotations by using spatial clustering and self-supervised scene flow to generate static and dynamic object proposals from LiDAR data. It then encodes the visual appearances of these proposals to distinguish static objects in the foreground and background. The main contribution is the method design that jointly uses camera, LiDAR, and temporal information to train existing 3D object detectors in an unsupervised manner.

Strengths

1. The paper is well-structured, with a clear abstract, introduction, methodology, experiments, and conclusion sections that logically flow from one to the next. 2. The use of figures and diagrams, such as Figure 1, effectively illustrates the process and contributes to the clarity of the UNION method. 3. The use of pseudo-classes based on object appearance for training classifiers is innovative, offering a new way to tackle multi-class object detection without relying on manual annotations. 4. The UNION method introduces a new approach to unsupervised learning by combining spatial clustering, self-supervised scene flow, and visual appearance encoding in a synergistic manner. This represents a creative fusion of existing ideas applied to the problem of 3D object detection.

Weaknesses

1. The claim of being 'the first to do unsupervised multi-class 3D object detection' requires scrutiny. It's noted that other works, such as the one by Wu et al. presented at CVPR 2024, also delve into unsupervised multi-class 3D object detection. If the paper is accepted, it would be prudent to revise this statement to reflect the current state of research accurately and avoid overstating the novelty of the approach. 2. The decision to solely utilize the nuScenes dataset for experiments raises questions about the breadth of the evaluation. The Waymo dataset, with its denser point clouds, might offer a better testbed for supervised tasks. Furthermore, the official multi-class metric of the Waymo dataset, which assesses the detection of vehicles, pedestrians, and cyclists, provides a more straightforward comparison to fully supervised detectors. It would be beneficial to consider a comparative analysis using the Waymo dataset to strengthen the paper's findings. Rebuttal may not have time for this work, but can be part of the future. 3. The observed zero AP for the cyclist class is concerning and warrants a deeper investigation. It is suggested that the discussion should be expanded to address potential solutions to this issue. While the less sample is a contributing factor, it's also essential to explore other possibilities, such as the high similarity in appearance between pedestrians and cyclists, which might lead to misclassification and the pedestrian labels actually being cyclist labels. [1] Wu, Hai, et al. "Commonsense Prototype for Outdoor Unsupervised 3D Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

See weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have provided a discussion on the limitations of their work. They acknowledge that the method makes implicit assumptions about the occurrence frequency of objects.