Thanks for the revised score and addressing concerns regarding novelty
Thank you for your thoughtful review and for recognizing the efforts we've made in addressing your previous concerns about evaluation. We're pleased that the additional experiments and the inclusion of mask guidance have helped clarify certain aspects of our work.
Regarding the novelty, we understand your reservations about combining several foundation models. However, we believe that the integration of these models in our context is not just a mere combination. In addition to utilizing foundational visual encoders and text encoders found in previous methods, our approach uniquely leverages image segmentation foundation models (such as SAM), image captioning foundation models (like BLIPV2 and Tag2Text), and CHATGPT. To the best of our knowledge, our method represents the first exploration of these foundation models in the realm of 3D scene understanding.
Regarding the similarity to ULIP, it is important to highlight some key differences that set our method apart. While the training of ULIP relies on the annotation of 3D objects to extract corresponding text features, this approach is not applicable within 3D scene datasets. Our method uniquely overcomes this limitation by employing image-to-text foundation models to generate relevant text descriptions, further refined using CHATGPT. Moreover, we acknowledge that our method, like ULIP, utilizes both text and image during the pre-training stage. However, it's essential to recognize the distinctions between those two methods. ULIP's instance-level contrastive approach serves well for tasks like instance discrimination but falls short for dense prediction tasks such as object detection and semantic segmentation. This limitation has been observed in various works across 2D and 3D domains [1, 2, 3, 9, 53]. There exists a non-negligible gap between instance-level pre-training as ULIP and our proposed scene-level dense pre-training method.
The novelty of our work lies in the development of self-supervised pre-training that is tailored and aligned with 3D object detection and segmentation. This alignment is achieved through the strategic integration of foundation models from both image and text modalities, forming a coherent framework that transcends previous limitations in the field.
We sincerely appreciate your feedback and the increased score. Your insights have been instrumental in refining our work, and we're grateful for your constructive engagement with our research.
[1] Xie, Zhenda, et al. "Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Wang, Xinlong, et al. "Dense contrastive learning for self-supervised visual pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[3] Liu, Yueh-Cheng, et al. "Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining." arXiv preprint arXiv:2104.04687 (2021).Liu, Yueh-Cheng, et al. "Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining." arXiv preprint arXiv:2104.04687 (2021).