OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images

Summary

This paper focuses on 3D open-world learning on classification. To address the limitation caused by existing CAD-rendered images in open-world 3D learning, it proposes to generate depth-aligned images from point cloud projected depth maps and diffusion models.

Strengths

1. Generating depth-aligned images to assist open-world 3d learning is interesting. 2. The method is evaluated on multiple 3D classification tasks.

Weaknesses

1. It seems that the generated textures among multiple views for a single object are not consistent. Would this largely influence the performance? 2. Generating images with diffusion models for image/3D understanding is a prevailing area [A]. The author should discuss existing works in the related works. Also, It turns out the method is not novel for me. 3. 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets. [A] Is synthetic data from generative models ready for image recognition?

Questions

Please refer to the weakness part.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

No.

Summary

This paper proposes OpenDlign, a framework for depth-based 3D understanding by aligning depth-image features through training on generated depth-aligned images, which overcomes the limitations of training on CAD-rendered images. Using point cloud data from ShapeNet, a contour-aware projection method is introduced to produce dense depth maps from the point clouds. To prepare the training data, an off-the-shelf depth-controlled diffusion model is employed to generate the corresponding RGB images from the projected depth maps. Additionally, techniques such as depth-specific text prompts and logit aggregation are adopted to enhance performance. Experiments demonstrate state-of-the-art results in zero/few-shot 3D classification, zero-shot 3D object detection, and cross-modal retrieval tasks.

Strengths

- The main idea of using generated depth-image pairs rather than CAD renderings to align a depth encoder with the image encoder is a reasonable design choice, enhancing generalization and robustness. - The experiments are very comprehensive. The proposed method appears to achieve state-of-the-art results across all the evaluated tasks, which is quite impressive. Ablation studies are also very thorough, revealing the contribution of each design choice. However, since I am not an expert in these tasks, I cannot fully confirm the true status of this model's performance compared to other state-of-the-art methods. - The paper is generally well-written and easy to follow.

Weaknesses

- The motivations for the major design choices in this work are questionable. They appear to be driven purely by the evaluation benchmark (on where all methods use point cloud as the input, and some methods convert point cloud into depth later), rather than focusing on training a good encoder specifically for 3D understanding from depth maps. If this is the case, the title could better reflect this by referring to point cloud-image alignment rather than depth-image alignment. Specifically: - Why not use mesh to directly render depth maps? If the starting point is depth-image alignment, there is no need to convert point clouds into depth maps, which is only meaningful if the evaluation benchmarks are based on point cloud inputs. The availability of real depth maps renders one of the major contributions, the contour-aware projection method, completely irrelevant to the topic of this work. - Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything? This approach should significantly reduce the computation overhead of dataset creation and enable training on even more diverse data. Again, it seems that the current design choices are purely driven by the point cloud-based evaluation benchmark. - In L152, the definition of inverse depth is incorrect. Inverse depth, also known as disparity, is 1/D. Depth ControlNet should take the normalized inverse depth (normalized into [0, 1]) as the input, rather than 1 - D.

Questions

Please refer to the questions in the weaknesses section.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Limitations have been discussed. I don't see any potential negative societal impact in this work.

Summary

The paper introduces **OpenDlign**, a novel framework for open-world 3D representation learning by leveraging depth-aligned images generated from a diffusion model. OpenDlign aims to enhance the realism and texture diversity in the 3D learning process, overcoming the limitations of CAD-rendered images. The method involves fine-tuning the CLIP image encoder with depth-aligned images, achieving superior performance in zero-shot and few-shot classification tasks. Experimental results demonstrate significant improvements over existing state-of-the-art models on various benchmarks like ModelNet40, ScanObjectNN, and OmniObject3D.

Strengths

- **Innovative Approach**: The introduction of depth-aligned images generated from a diffusion model is a novel and effective way to enhance texture diversity and realism in 3D learning.   - **Significant Performance Gains**: The experimental results show substantial improvements in zero-shot and few-shot classification tasks, highlighting the effectiveness of the proposed method.   - **Detailed Methodology**: The paper provides a comprehensive description of the methodology, including the contour-aware projection method and the multimodal alignment framework.   - **Broader Impact Considerations**: The authors discuss both positive and negative societal impacts, demonstrating awareness of the potential implications of their work.   - **Reproducibility**: The paper includes sufficient details on the experimental setup and training process, enhancing the reproducibility of the results.

Weaknesses

- **Limited Dataset for Depth-Aligned Images**: The generation of depth-aligned images is limited to the ShapeNet dataset, which might restrict the generalizability of the results to other datasets.   - **Computational Resources**: The approach requires significant computational resources for generating depth-aligned images and fine-tuning the model, which might not be easily accessible to all researchers.   - **Potential Biases**: The paper acknowledges biases in the CLIP and ControlNet models but does not provide detailed strategies for mitigating these biases in the generated depth-aligned images.

Questions

- More applications: could the authors provide application study in other 3D OV Understand tasks like open-vocabulary 3D object detection methods[1, 2]   - Can the authors provide more details on the computational cost and feasibility of generating depth-aligned images for larger datasets?   - How do the authors plan to address the potential biases in the depth-aligned images generated by the diffusion model?   - Are there any plans to extend the approach to other types of 3D datasets beyond ShapeNet?   - Could the authors elaborate on the scalability of the method when using larger CLIP models or other vision-language models? [1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3. [2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have provided discussions about the limitations.

Summary

The paper introduces OpenDlign, a novel approach to 3D open-world learning. Traditional 3D learning models struggle with unseen categories and typically rely on CAD-rendered images, which lack texture diversity and realism, leading to suboptimal alignment and performance. OpenDlign leverages depth-aligned images generated from a diffusion model to enhance texture variation and realism. It seems OpenDlign achieves superior zero-shot and few-shot performance across diverse 3D tasks.

Strengths

1. Overall the core idea of this work makes sense, i.e., instead of using CAD rendering, it uses an off-the-shelf diffusion model to convert 3D assets to 2D images, This increases the diversity of training and should be able to improve the generalisation to unseen data. 2. The performance of the proposed method seems strong.

Weaknesses

1. Although the method shows a strong performance improvement when using ShapeNet for training, such a gain is not clear when using the "Ensemble" dataset for training. For example, in Table 1, TAMM-PointBERT (+dlign) is only higher than TAMM-PointBERT by 1.2% on ModelNet40 Top1, and even worse than on ModelNet40 Top5. Similarly, when tested on the ScanObjectNN, OpenShape-SparseConv (+dlign) is worse than OpenShape-SparseConv over all metrics. Does it mean the proposed method is not important/beneficial when the training data is large and diverse enough? 2. Probably I missed something, but it seems the authors did not mention the name of the diffusion model they used in the paper. As shown in Figure 2, the "Multi-View Depth-Aligned Images" are also fed into CLIP Image Encoder. However, how to ensure multi-view consistency is still an open problem for diffusion models. Usually, such generated multi-view images will have obvious inconsistency. How did the authors address this problem? This might be a contributing factor to the performance drop mentioned in point 1, where methods show decreased performance when using OpenDlign. 3. Section 3.2 indicates that the proposed framework generates a single set of multi-view images for a given 3D asset. It seems logical to generate multiple sets of multi-view images for a 3D asset to enhance training diversity and reduce dependency on image appearance. Is there a specific rationale behind limiting the generation to a single set of multi-view images?

Questions

Overall the reviewer is concerned about the point 1 and 2 mentioned in the Weaknesses. The reviewer may adjust the score based on the authors' response.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Further reply

Thank you to the authors for the discussion and additional experiments provided during the rebuttal period; most of my concerns have been addressed, I hope the authors can update the comparative experiments with OV-3DET and CoDA in the final version, as this knowledge will be beneficial to the related fields of the community. Totally speaking, I am inclined to accept this paper and decide to raise my rating to 7. Gook luck :)