Summary
This paper addresses the challenge of open-vocabulary 3D semantic segmentation by utilizing 3D geometric features, 2D semantic embeddings, and text modality. The proposed approach adapts the ODISE method to the 3D domain, aiming to distill open-vocabulary semantic segmentation knowledge from a pre-trained text-to-image denoising diffusion model to a 3D segmentation model. Initially, an input point cloud is fed into a 3D encoder-decoder segmentation network, producing point-wise geometric features. Simultaneously, a pre-trained visual-language diffusion model generates 2D masks and embeddings from posed images of the same scene, conditioned on the 3D global feature of the 3D branch’s encoder. Unlike the ODISE method, an implicit $3D$ captioner is introduced to produce geometry-aware 2D masks while also distilling information from the 2D branch network to the 3D encoder. To further regularize the 3D network, a distillation loss ($\mathcal{L}_{mask}$) is applied to the 3D mask embeddings, derived from the per-point features and the 2D masks back-projected to the point cloud as 3D binary masks. By obtaining ground truth mask features from a pre-trained CLIP model, the 3D masked embeddings are aligned with the image-text joint embedding space, through a cosine similarity loss. This alignment leads to more coherent segmentation results and enhances the model's open-vocabulary capabilities. Finally, the per-point features are combined with the pseudo mask 2D features (formed by the back-projected 3D mask and 2D mask embeddings), resulting in a fused per-point representation that incorporates the geometric information from the 3D segmentation network and the semantic open-vocabulary capabilities of the 2D branch. The approach is evaluated on three semantic segmentation benchmarks (ScanNet, ScanNet200, and S3DIS) and demonstrates superior performance compared to competing methods.
Strengths
The XMask3D effectively aligns 3D geometric features with 2D and textual modailities through knowledge distillation from visual-text joint embedding spaces inherent in the pre-trained 2D denoising UNet and the CLIP model. As evident by the ablation, the implicit 3D captioner is a crucial step in the overall pipeline, and it outperforms vanilla text conditioning or the implicit 2D captioner of ODISE, in both base and novel semantic categories. Moreover, the 2D-to-3D mask regularization is also essential, since it significantly improves the accuracy of the proposed method esp. in novel categories. This justifies the need for this additional distillation step from the CLIP joint space, to further enhance the open-vocabulary capabilities of the XMask3D method. Finally, the discussion on modality fusion, both in the main paper and supplementary, is highly appreciated. By dissecting the method and providing qualitative and quantitative results for each step, the authors make it easier for readers to understand and gain intuition about the presented approach.
Weaknesses
While the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).
Questions
Following the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($\mathcal{L}_{seg}^{2D}$, $\mathcal{L}\_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?
Limitations
Yes, the authors have discussed the method's limitations in detail in Section 4.4.