VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation

Summary

The paper proposes an efficient conditional 3D generation via voxel-point progressive representation. More specifically, a voxel semantic generator and a point upsampler have been created to achieve efficient generation on multi-category objects. To verify the effectiveness of the method, extensive experiments with SOTA results are achieved.

Strengths

1. The paper is well-written and well-organized. 2. Extensive experiments are conducted, and impressive results are obtained.

Weaknesses

1. It seems that the paper can address point upsample task instead of upsample completion task, since upsample refers to generate dense points while completion refers to synthesize new points given partial scans. 2. It would be better to show some ablation studies on the network architectures such as removing or replacing specific components to see how they affect the performances. 3. It would be clearer to indicate Tab .3 in line 234 for the corresponding quantitative results.

Questions

1. To reduce the degree of confidence when generating voxels because of the high masking rate, Gaussian noise is added to the prompt embedding.The question is instead of adding Gaussian noise, how about directly sampling smaller masking rate? 2. How is Eq. (2) obtained? It would be better to provide some hints on the calculation.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This work proposes to use a voxel-point progressive representation for efficient 3D generation, and it proposes a few architectures for different applications, including generation, editing, upsampling, and pre-training. Based on the reported results, the proposed method could generate various 3D shapes and could achieve competitive classification results.

Strengths

1. The work proposes a voxel-point progressive representation, which could provide good 3D generation results on the ShapeNet dataset as shown in the reported experiments. 2. Many modules, such as 3D VQGAN, Voxel Semantic Generator, Grid Smoother, and Point Upsampler, have been proposed.

Weaknesses

1. The proposed method could only generate 3D shapes belonging to the categories of the ShapeNet dataset. It does not show any 3D shape results from unseen categories even with the help of CLIP. 2. Point-Voxel representation has been broadly studied in previous methods, such as [A-C], which has already been proven to be an efficient representation for 3D point cloud analysis. 3. The classification results for both the ScanObjectNN and ModelNet40 datasets are very saturated. Also, the improvements seem to be very incremental. [A] Liu, Z., Tang, H., Lin, Y., & Han, S. (2019). Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32. [B] Zhang, C., Wan, H., Shen, X., & Wu, Z. (2022). PVT: Point‐voxel transformer for point cloud learning. International Journal of Intelligent Systems, 37(12), 11985-12008. [C] Liu, Z., Tang, H., Zhao, S., Shao, K., & Han, S. (2021). Pvnas: 3d neural architecture search with point-voxel convolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8552-8568.

Questions

1. Is the proposed method capable of generating 3D shapes of novel categories? 2. Is the proposed method capable of generating complete point clouds given a partial point cloud? 3. Does the proposed method demonstrate robustness when presented with noisy point clouds as input? 4. Is the generated 3D shapes caused by overfitting the ShapeNet dataset? Maybe the authors could try to retrieve the most similar shape in the ShapeNet dataset to prove its generation ability.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

yes

Summary

Authors propose an approach to generate 3D point clouds of objects with an image or text description as input. Authors use a pre-trained CLIP model to generate text/image embeddings and use this to first generate features in voxel space (Voxel Semantic Generator). These voxel features are then decoded into a coarse voxel grid. Authors then convert the voxels to point clouds and use a smoothing network to obtain a uniform point cloud. Authors use a transformer to then convert this point cloud into a detailed 3D shape (also point cloud). Authors show qualitative and quantitative comparison with relevant baselines (CLIP-Sculptor, CLIP-Forge and Point-E). Authors also show several applications like text and image conditioned 3D generation, shape editing and completion.

Strengths

+ Ideas presented in the work are technically sound. + Presented results outperform competing baselines. + Authors submitted code. Although I did not run it but this is still appreciated. Code will also be released upon acceptance. + Paper is mostly well written. See comments below for minor improvements.

Weaknesses

[Technical] 1. How is the proposed Voxel Semantic Generator different than “CLIP-Conditioned Coarse Transformer” from CLIP-Sculptor? They are essentially doing the same thing, 1. Learn a voxel based encoding of 3D shapes and 2. Use a transformer to learn how to unmask the 3D features conditioned on a prompt. Can authors clarify this better? 2. L175-178: Since the GT for Grid Smoother is generated using furthest point sampling, why can’t we use FSP at inference time to smoothen the coarse voxel grid from the 3D VQGAN decoder? Why do we need to learn a neural network? How useful is adding KL loss over Chamfer loss? 3. L180: The output of the grid smoother is a point cloud (I assume this because it is trained with a Chamfer loss). L169 mentions that “point tokens” are masked which are then unmasked by the transformer. What exactly are the “point tokens”? Is it just the positional encoding of points? Fig.2 (b) mentions “semantic tokens” what does this mean? It is not explained in Sec 3.3 (Point Upsampler). 4. Eq. 1: How useful is the occupancy loss over MSE? Is this critical? Please add an ablation study in supp. mat. to support the proposal. [Minor] - Fig. 2: Please add symbols used in the text to Fig.2. This makes it easier to follow the text and map various components of the pipeline. - Add dimensionality of each embedding/latent code along with the symbols, eg: C_{pr} \in \mathbb{R}^{<whatever>\times<whatever>}. This significantly improves clarity and readability. - L120: How is the mapping performed? How is the codebook generated? If an existing work is used, please add citation here? This is present in the supp. mat. to an extent. Add a pointer so that the reader knowns where to look. - L140: Just curious, how important is the scheduling of the mask fraction? L147: Is there significant performance difference at inference time between direct and multi-step prediction? - L234: Table number is missing. - Please avoid violent objects like guns for showing qualitative results. It is understandable that it is sometimes necessary to demonstrate performance but whenever possible, let us try to avoid this.

Questions

Overall I'm positive about the work. There are some concerns about the novelty/necessity of some components (see above) and it would be great if authors can clarify these.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussion on Limitations and Broader Impact is not included in the main paper but it is present in the supp. mat. Authors are encouraged to add some discussion on limitations and future work in the main paper as it allows the community to better use and build upon your research.

Summary

The VPP proposes a model for 3D generation. It utilizes both point-based and voxel-based representations. Voxel-based representations are used to generate the coarse tokens, and the point-based one further improves the result. Both of which are pretrained with MAE-like self-supervised method. The proposed method applies to image-to-point, text-to-point, and point completion. The propsed method is novel and effective. However, some parts are not presented clearly.

Strengths

1. The limitations of existing methods are well analyzed. 2. The proposed method is carefully designed and performs well. 3. The proposed method applies to more downstream tasks than exsiting methods. 4. The experiments show the effectiveness of the proposed method.

Weaknesses

1. The inference efficiency in Table 1 only presents hours or seconds, it is recommended to provide the exact time. Totally, the paper is not well presented. Some details are not shown clearly. To be specific, 2. The tokenizer for points upsampler is not described. What is it structure and how to train it? 3. It would be better to refer to some important symbols in Figure2. For example, prompt embedding in Figure 2 should also be marked as C_pr. 4. The GAN's structure and loss in 3D VQGAN are not introduced. 5. As described in Section 3.2, the mask voxel transformer will be forwarded multiple times for better quality. It should also be commented in Figure 2 or 3 for better presence. Some typos: 6. In about line 151: "when the inputs of the mask voxel transformer are grid tokens with high masking fraction and prompt embedding we find that this will lead to the semantics of the predicted grid tokens being much more biased toward the prompt embedding". I think the authors may miss punctuations.

Questions

1. I am not sure what parallel decoding in Figure 3 means. 2. When training the voxel semantic transformer, part of voxel tokens are masked, as shown in Figure 2a, and the masking ratio is about 64%. While all tokens are masked voxel tokens during inference. Is there a gap between training and inference?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

Not applicable

Summary

The paper proposed a text-driven point cloud generation model that can be used for various downstream tasks such as generation, editing, completion and pretraining, while being very efficient. The method largely follows Muse [2], but adapted it to 3D point clouds. The model consists of multiple components, which are trained individually. First, a VQGAN is trained on the voxelized shapes, which embed a shape to a latent grid. Next, a masked transformer is trained on the latent grid to model its distribution, using CLIP feature as semantic conditioning. For the decoding stage, a grid smoother network moves the voxel centers to more homogeneous locations, while finally another set of point cloud VQVAE and masked transformer is trained to upsample the coarse point cloud to high resolution point cloud. The paper also demonstrated that the proposed components can be used in whole or in part to tackle a wide range of tasks.

Strengths

* The proposed method is able to handle a wide range of tasks such as generation, editing, completion and pretraining. * VPP significantly outperforms baseline methods on the main task of shape generation. * The proposed method is computationally efficient compared to optimization-based methods such as DreamFusion.

Weaknesses

* The components of the proposed method are not new -- they are mostly borrowed from previous works such as CLIP, MUSE and Point-MAE. * The method contained a large number of stages, making it potentially difficult to implement or improve upon. * Unlike diffusion-based methods such as DreamFusion, Dream3D, Magic3D, the proposed method does not generate surface or texture. It is also not clear if the proposed method is able to do zero-shot generation like these methods. * The proposed method is only trained on ShapeNet, which is relatively small and less diverse compared to larger datasets such as Objaverse. It is not sure if the method is able to scale to larger datasets.

Questions

* What is the benefit of using voxel as intermediate representation compared to directly encoding the point cloud? * Missing table reference on L234.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations and societal impacts are adequately addressed in the supplemental material.

Thanks for the rebuttal

I would like to thank the authors for the clarification and the extra experiment on Objaverse. I have raised my rating as all of my concerns are resolved. The reason that prevents me from giving any higher rating is that, despite the model seems to be very powerful, its zero-shot generation capability appears to be very limited, which will limit its usefulness.

Post rebuttal update

Thanks authors for the rebuttal. It addresses my concerns and I keep my positive rating of the work.

Further Discussion

Dear Reviewer A5WF, Thanks again for your valuable comments and suggestions! The Author-Reviewer discussion is coming to an end, and we hope that we have addressed all of your concerns. Please, let us know if you have any follow-up questions or concerns. We will be happy to answer them. Best Regards, Authors

Further Discussion

Dear Reviewer THuC, Thanks again for your valuable comments and suggestions! The Author-Reviewer discussion is coming to an end, and we hope that we have addressed all of your concerns. Please, let us know if you have any follow-up questions or concerns. We will be happy to answer them. Best Regards, Authors