Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Summary

This paper proposes a conditional 3D generation method by pre-aligning features of different modalities when training a 3D AE. After that, a latent diffusion model is applied to generate latent vectors for 3D decoder conditioned on text/image. The results quantitatively and qualitatively shows that the proposed method is able to achieve impressive results.

Strengths

1. The results show that this method is able to achieve multiple tasks, which is good and important for a generation-related paper. 2. The main figure is easy to understand and follow, making this paper easier to read. 3. The core idea in this paper looks original.

Weaknesses

1. No qualitative results show the diversity of the proposed method. Such a result is important to a generation-related paper. 2. The concept of pre-alignment is not very convincing. Other conditional generation methods are also aligning text/image to the shape latent space by either pulling their representations together or learning diffusion models. Further discussion on how this concept is better than others should be added.

Questions

1. If you have paired data for text-image-shape alignment training, why not just directly train a latent diffusion model conditioned on image/text to generate 3D objects like SDFusion[1]? In contrast, if you already aligned text/image/shape features in latent space, is it still necessary to use a heavy conditional generator such as latent diffusion to achieve your tasks? It feels like you only need one of these two techniques. 2. In your qualitative comparison, you mentioned that 3DS2V generates more details of 3D shapes and discontinuous surfaces and noisy results. However, such results are not seen in their paper, I wonder the reason why they fail to produce good results in Figure 2 and 3. Is it because you re-train their model on your dataset or because you directly use their pre-trained checkpoint or other reasons? More discussion and insight should be addressed. 3. You pre-aligned the text/image/shape feature first and use text/image feature to train a latent diffusion model to achieve conditional generation. However, as far as I know, if the input condition and the target feature are too similar when training the diffusion model, the diversity of the generation result would be limited and the purpose of using the diffusion model would disappear. How do you avoid such a situation from happening? 4. Please answer the problem mentioned in the weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

yes

Summary

This work is about 3D shape generative model focused on image-conditioned and text-conditioned generation. The authors aligned the latent space of a shape autoencoder to CLIP's image encoder and text encoder. Then generative diffusion models are trained on the aligned latent space. This enables shape generation given image or text as conditional input. The authors showed some good results in both tasks.

Strengths

The authors showed some good generation results for both the task of image-conditioned generation and text-conditioned generation. The writing is also clear and easy to follow.

Weaknesses

1. The autoencoder network (Fig 1 a) is similar to the network used in [63]. Thus the performance boost shown in Table 1 seems to be because of CLIP. This should be emphasized or ablated somewhere in the main paper. 2. Following the above comment (Table 1), the authors only compared with "Learned queries" results from [63]. According to [63], another design "Point queries" achieved better results than both this work and "Learned queries". 3. It would be better if the authors can show some visualization comparisons of the autoencoding results. 4. L136, the length is (1+L_i) instead of L_i. This is explained in later sections (L217) but is still causing confusion. 5. Another difference with respect to [63] is, when training the diffusion models, this work used a unet-style transformer [4] instead of a simple stacked self attention network. 6. Some pieces of writing can still be improved. For example, when talking about a design or an equation, we should mention why we are doing this or discuss some insights behind this.

Questions

See the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This work proposed a method for image-conditioned and text-conditioned shape generation. The authors combined several components (3DS2V-style shape autoencoding network, clip and unet-style transformer denoising network). All these components are not proposed by this work which weakens the novelty of this paper. However, I still believe the authors delivered some good results in shape generation. I hope the authors can clarify my concerns mentioned in the weakness section.

Summary

This paper proposed a conditional generation model which aims to solve the alignment issue in image-to-shape or text-to-shape generation. The key idea is to learn a aligned representation among 3D shapes, images, and texts. To achieve that, the author proposes SITA-VAE with contrastive loss to force the shape's latents to be aligned with the pretrained vision-language model. After that, a LDM is trained to learn the diffusion process in the latent space. In the test time, it follows the previous works to use classifier-free guidance to perform conditional generation. The proposed method is evaluated on ShapeNet and Cartoon Monster 3D shapes.

Strengths

* Propose to learn a aligned space for 3D shapes, images, and texts with 2D vision-language model. This essentially leverages the abundant 2D data into 3D modeling. The scarcity of paired data is one crucial reason why conditional 3D modeling does not perform as well as its 2D counterpart. * The conditional generation results look great and align with the inputs when compared with the baselines. * It is surprised that the aligned space does hurt the reconstruction results.

Weaknesses

* Novelty is somewhat limited. The first stage model is mostly based on 3DILG and the multi-modality conditional generation has been explored in previous work such as SDFusion. * Scalability is an issue as the model need paired 3D-image-text data to work. * Some parts of the writing and figure are misleading. The choices of the metric is problematic in Table 2. Please see the "Questions" for the details.

Questions

1. The results in Fig. 2 seems a bit too sharp and I wonder if the model is memorizing the outputs. Can the author shows the nearest neighbor sample from the datasets? 2. In figure 1, the model is outputting occupancy but the final result is denoted as "mesh". I think the output should be occupancy and the mesh is obtained via marchingcube, right? Learning to generate mesh directly is very different from generating occupancies. The writing also use the word "mesh" and it is a bit confusing. 3. In Table 2, the use of SITA to evaluate the performance of conditional generation is unfair compared to other baselines since the LDM is trained on the space learned with SITA.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Authors have address the limitations.

Summary

This paper proposed a VAE-based text-to-3D shape generation method. The authors designed an alignment-before-generation approach to narrow the gap between 3D shapes and the 2D or text condition. They first train a Shape-Image-Text-Aligned Variational Auto-Encoder to align the representations between the 3D shapes and the 2D or text inputs. Then, the use latent diffusion model to denoise the shape embeddings to match the conditions. The overall presentation is good and the experiments are extensive.

Strengths

1. The narratives are good and sound. 2. The SITA-VAE helps to align the representations between the three modalities. 3. The generation only requires some denoising steps on the latent embeddings. 4. The experiments are extensive and the authors provide sufficient visual demos and show the effectiveness of the method.

Weaknesses

1. It seems that shape-image-text align training requires a lot of paired shape-image-text data, which could be a huge challenge in the generalization to new categories. 2. The experimental implementation is not clear enough and could be improved.

Questions

1. Since this method is trained with ground truth 3D data, which limited its generalization ability, would it be better to show some failure cases when the condition fails to lie in the aligned representation space?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The author mentioned the limitation of the requirement of the 3D training data, which could be a huge problem in the generalization of this method.

Following comment

Thank the authors for the hard work! The feedback has addressed most of my concerns. I will raise my rating to 5.