OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

Summary

The paper proposes a new system for learning a joint embedding space for text, images, and 3D shapes (point clouds). It starts from pre-trained CLIP embeddings for text and images, and learns one more embedding function for 3D shapes in a self-supervised manner, using a contrastive loss. Several additional improvements are proposed, including combining several datasets, cleaning some of those including via automated re-labelling of Objavrese, and optimisation of the 3D embedding neural network architecture. Because of these improvements, the new embedding performs significantly better than prior works on strand tasks such as "zero-shot" classification.

Strengths

* The paper contributes sound incremental work on learning multi-modal embedding functions for text, images and 3D shapes. Various engineering dimensions are considered (model scaling, data diversity, scaling and quality, etc) and some insights are provided on how each of these can be improved, at least compared to current baselines. I expect these findings to provide useful guidelines for future work in this area. * The paper does contain some interesting suggestions on how to improve datasets such as Objaverse to train multi-modal embeddings. The scheme where captions are assessed via GPT-4 and then replaced using BLIP if needed is useful.

Weaknesses

* The paper is mostly about good engineering, but there isn't a lot of very deep technical innovations, or qualitatively surprising findings. The main finding, in fact, is that this is an area where scaling is still very limited (mostly due to the lack of suitable training data), and that, thus, scaling is where most of the low-hanging fruits can be had. This is a good but not very surprising message. * The model is trained on synthetic datasets and looses much of its edge when applied to real data, compared to prior models like ULIP that are otherwise suboptimal when tested on synthetic data (e.g., Figure 5, right panel). * Likewise, there isn't a very clear dominance of the two proposed backbones, SparseConv and PontBert: their performance swings significantly depending on the testing data in Table 2.

Questions

* Filtering bad positive samples in contrastive the loss using the existing text and image embeddings (line 213) is simple but also likely useful, however it is not clear to me if this idea is ablated in Table 3. Can you clarify? * Why SparseConv and PointBERT are so different in performance depending on the test dataset? Is there a recognisable pattern?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The paper does address some limitations, particularly in the discussion and conclusions where they are implicitly recast as "future work". There is no discussion of ethics, although this does not seem to be an issue for this paper (notoriously, though, dataset like ShapeNet had issues with copyright, and I don't believe ObjaVerse is necessarily immune from such controversies either).

Summary

The paper studies the problem of multi-modal learning of text, image and shapes. Shapes are represented by pointclouds and the learning is driven by the standard contrastive loss. The main technical difference of this work to prior ones lies in the scale of data and corresponding training strategies. The authors train their model on a much larger scale of data which pools Objaverse, ShapeNet, 3D-Future and ABO. To handle the noise in the text (mainly from the less-curated Objaverse), the authors use several large pretrained models to clean up the text and make it better aligned with the shapes. Hard-mining is also leveraged to make the model train better under class imbalance. With these techniques which make the model scales better with data size, the proposed model achieves the new SOTA performance across several relevant tasks and datasets.

Strengths

1. The model in this paper achieves significantly better results than previous SOTA in zero-shot shape classification. It also demonstrates great qualitative performance in shape retrieval, shape interpolation, shape captioning and has good potential in other cross-modal applications as well. Such model will be practically very useful. 2. The benchmark and the findings in the paper about different models' scaling performance are valuable information to the community and future research along this direction. 3. The paper is well-written and easy to follow.

Weaknesses

1. The evaluation of the zero-shot shape classification on Objaverse-LVIS might not actually be zero-shot. Based on the description in L235, even the "Ensembled (no LVIS)" can include LVIS categories, because seemingly the author only excludes the exact evaluation samples from the training set. Ideally, all shapes from the evaluation *categories* should be removed from the training set. This makes the real zero-shot generalization performance of the model potentially lower than reported and the overall comparison on this test set less informative. Besides, ModelNet40 and ScanObjectNN can also have overlapping categories with the training set. (If the authors think category overlapping does not violate their definition of zero-shot, it should be clearly stated so / "zero-shot" should be clearly defined at the beginning.) 2. The linear probe results show the proposed model performs similarly to ULIP-retrained on ModelNet and ScanObjectNN. The authors try to explain it with in-category sample bias (what does this mean?) and domain gap dominance, without providing any evidence for these hypotheses. My concern here is that, given that OpenShape performs much better than ULIP in shape classification (requires both shape and language rep) but not linear probe (only requires shape rep), the actual underlying reason could be that the shape representation is of similar quality when tested on out-of-domain, and the real difference between these models lies in the text representation. I believe it would be quite beneficial to provide more analyses on this result, as this is central to the "shape representation learning" story of the paper. For example, such analyses could be visualizing and comparing the latent structure of the shape representation; or simply performing shape-latent based NN query and evaluate the distance between queried shapes; or measuring the shape latent's alignment with image latent, to name a few. 3. The limitation of the model is not discussed at all. This is a very important aspect for readers to thoroughly understand the contribution of this paper.

Questions

Please see the weaknesses for my main questions and concerns. Overall I like the paper as it presents useful findings for the community and the model's performance is strong. But I think more analyses and discussions are necessary to solidify some experiments and the overall claim. Another relatively minor question about the result: In Table 2, the retrained ULIP achieves 26.8% top 1 while OpenShape achieves 43.4%. The authors attribute this to "text enrichment, backbone scaling, and other training strategies" (L243). However, it is shown in the ablation that removing text enrichment leads to 6.4% drop and hard mining leads to 1.4% drop. Does this mean the main improvement over ULIP comes from the larger number of parameters/better backbones as the overall drop is ~17%?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations and failure cases are not discussed and should be included.

Summary

In this paper, the authors propose a joint learning framework for multi-modal representations among text, image, and point clouds. Specifically, the authors fix the pretrained CLIP language and image encoder, and align the point cloud representation by the proposed multi-modal representation alignment technique. Then, to enlarge the pretraining dataset, the authors introduce text filtering and enrichment technique to annotate point clouds without descriptions. The proposed OpenShape is pretrained on an large-scale ensembled dataset, and the experimental results demonstrate that OpenShape has promising zero-shot classification ability and shape retrieval ability.

Strengths

1. The proposed method fully leverages the large-scale point cloud datasets to align the point cloud backbone with frozen pretrained CLIP encoders, thus obtaining open-world point could recognition and shape retrieval ability. 2. The proposed text filtering and enrichment technique is effective to preprocess large-scale point cloud datasets. 3. The overall writing is polished. 4. The ablation study is extensive and critical.

Weaknesses

My major concern is the fairness in the experiments (i.e., table 2). As the authors stated in section 3.1, the chosen vision-language encoder is OpenCLIP ViT-G-14, which is an extremely large backbone and obtains much better feature representation ability. Therefore, the authors should list the point cloud backbone and pretrained vision-languge encoders concretely in the table, and make fair comparisons if possible.

Questions

See the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not state the limitation of this method. Instead, the authors present the future direction of the proposed OpenShape, i.e., part-level information and synthetic-to-realistic domain gap. These directions can be seen as the limitation of this work.

Summary

This work learns a multi-modal representation among text, images, and point clouds based on a scaling 3D dataset. The results demonstrate remarkable performance in point cloud zero-shot classification, retrieval, and captioning tasks. In order to construct this dataset, the authors combined four commonly used 3D datasets while filtering and enhancing text prompts. Additionally, they introduced an offline hard negative mining strategy to improve the efficiency of the training process.

Strengths

1. This paper collects 0.8M text-image-point cloud triplets from four popular 3D datasets and does text cleaning and enrichment to get high-quality prompts for each object. 2. This work proposes a hard negative mining strategy to improve joint representation training efficiency and performance. 3. The trained representation shows surprising performance on zero/few-shot classification.

Weaknesses

1. The major problem is an insufficient novelty compared with ULIP. The pertaining method, triplets construction and downstream tasks are very similar to ULIP. The primary distinguishing point of this work is the text prompt cleaning and enrichment flowchart, as well as the offline hard negative mining strategy employed. However, the latter strategy lacks an ablation study to provide supporting evidence. 2. The motivation behind this work is not clearly articulated. While the aim is to develop a scaling-up 3D representation, the purpose of this representation is unclear. Unlike ULIP, which focuses on enhancing current 3D backbones through developing a new pipeline, this work attempts to scale up text-to-3D pairs through text augmentation that is still limited by the insufficient number of 3D objects. 3. There is a lack of exploration into the downstream applications of pre-trained representation to counterparts in 2D. This work is more suitable for submission to the dataset and benchmark track.

Questions

- The main contribution of this work is the datasets ensemble. Will the authors release the collected dataset and pre-trained 3D representation, similar to ULIP? - How will multiple object combinations and scene models be addressed in the Objaverse dataset? Will such data be ignored? According to Table 2, pre-training on the assembled dataset has a negative impact on the ScanObjectNN dataset, indicating overfitting in the Objaverse dataset and a degradation of the transferability between the simulation and real domains. - Section 3.4 appears to be an experimental setup and should be placed in the experiments and analysis chapter. - Lack of related work: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No needed

Summary

OpenShape explores scaling-up strategy for learning joint representations of texts, image, and 3D point clouds. It proposes to construct larger-scale 3D datasets, filter and enrich paired texts for pre-training, which achieves SOTA results on zero-shot 3D classification and retrieval benchmarks.

Strengths

1. The zero-shot performance of OpenShape is impressive, surpassing existing SOTA methods with large margins. This demonstrates the significance of the method. 2. The retrieval visualizations are interesting and clearly illustrate the embedding ability of OpenShape. 3. The authors re-train exiting methods under new settings, which is good for a fair comparison.

Weaknesses

1. OpenShape utilizes GPT-4 for textual-level processing, which however is expensive to access. How about the performance using a more affordable GPT-3 (like PointCLIP V2) ? 2. I'm curious about how OpenShape can be incorporated with non-parametric 3D network Point-NN (or its parametric derivative Point-PN) for zero-shot learning? Starting from Non-Parametric Networks for 3D Point Cloud Analysis, CVPR 2023

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Fix the format error for the first response

**The difference between the proposed method and ULIP** While we follow the multi-modal contrastive learning framework for representation alignment as used in many previous works (not just ULIP), our objective is to learn a useful 3D representation to enable open-world 3D shape understanding by scaling up the pretraining. Here are some main differences: (1) ULIP mainly experiments on small-scale datasets (e.g., ShapeNet), while we aim to pretrain on a large-scale ensembled dataset and address the arisen challenges. The experiments show that even if ULIP is trained on our ensembled dataset, it’s still less effective than our method. (2) ULIP utilizes the original meta information (e.g., shape category and name) of 3D datasets for training, while the meta information may be missing or noisy when scaling up the dataset. We thus propose several strategies to filter and enrich text descriptions for the 3D models, which is proved to be effective and critical. (3) ULIP utilizes the off-the-shelf 3D backbones with default settings. We explore the influences and scaling laws of 3D backbones when pretraining on a large-scale dataset, which is proven to be critical when learning a scalable representation. (4) We propose a hard negative mining module to address the severe class imbalance issue of the ensembled 3D dataset. The module is optional and can bring additional performance gain (as ablated in Table 3). Moreover, it can enable more efficient pretraining. We observe a ~2x speed up when using hard negative mining. (5) In addition to the supreme performance over ULIP (both the original version and retrained version) on zero/few-shot classification, we showcase a lot of cross-modal applications enabled by our powerful representations, which are missing from the ULIP paper. The applications include: (a) image and point cloud input 3D shape retrieval (Figures 1, 8, S1, S2), (b) text input 3D shape retrieval (Figure 9), which supports fine-grained subcategories, attributes, and their combinations; (c) point cloud captioning (Figure 10 and S3); (d) point cloud conditioned image generation (Figure 10 and S4). We kindly refer the reviewers to the supplementary material for more exciting results. Note that the ULIP paper only shows some simple cases of image-based retrieval (coarse category level), and does not demonstrate the capability of open-world understanding and supporting a wide range of downstream applications.