Summary
This study represents the pioneering effort to utilize 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds. They introduce Seal, an innovative framework specifically designed to extract informative features from sequences of automotive point clouds. With its emphasis on scalability, consistency, and generalizability, Seal effectively captures semantic-aware spatial and temporal consistency, enabling the generation of highly valuable insights. Finally, they clearly demonstrate their superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for various downstream tasks.
Strengths
1. The paper is well-written and easily comprehensible.
2. To my knowledge, this is the first attempt to utilize large-scale vision models for aiding 3D point cloud segmentation. The authors have also put in tremendous effort, completing this work within just a short span of one to two months.
3. Conducted extensive experiments and achieved satisfactory results on various segmentation datasets.
Weaknesses
1. Strictly speaking, the approach in this paper is not considered unsupervised pretraining as it utilizes large models that rely on additional data. Most methods in Table 1, however, are unsupervised pretraining. Nevertheless, this distinction is not crucial, as the primary concern is to improve model performance, which is achieved through effective methods.
2. I believe that the approach presented in the paper has not fully distilled the knowledge of large 2D models. When using the complete dataset for both pretraining and fine-tuning, the information gain from distilling large models is not significant. Why do I say this? In the method employed in this paper, the large models primarily provide semantic supervision signals. If full semantic labels are given during fine-tuning, the information gain is limited. Although Table 2 shows significant improvements for some full dataset cases, this is mainly due to the use of semantic features in distillation rather than simple semantic pseudo-labels. The advantage of distilling large 2D models is that I can use more image-lidar pairs during pretraining. I believe that if additional image-lidar pairs are used, such as on the nuscenes dataset, there will be further improvements. I also hope to see similar experiments, such as pretraining with ten times the number of image-lidar pairs compared to nuscenes.
3. The method proposed in this paper is expected to be very general, so it should yield better results for other 3D perception tasks (e.g., 3D object detection).
Questions
The questions are all in the weaknesses section.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Limitations are all in the weaknesses section.