DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

Summary

This paper proposes DiT-3D, an extension of DiT for point cloud data to achieve point cloud generation. With the design of patch embedding and 3D window attention, the proposed model is able to reduce the computation cost. Also, the proposed model is able to directly leverage pre-trained 2D DiT model by fixing most layers and only finetuning several layers, which can reduce the required training time. The results quantitatively verify the effectiveness of the proposed method.

Strengths

1. The proposed method out-performs existing SOTAs on the unconditional point cloud generation task in different evaluation matrices. 2. The paper is easy to follow and understand.

Weaknesses

1. The proposed DiT-3D looks like the original DiT with several modifications (simply changing existing 2D techniques into 3D version). Specifically, it seems like an ECCV 2022 paper [1] has already addressed a similar design of window attention, this paper is just using such a technique on a different task. Therefore, I think the novelty is limited, and more insights should be explained. 2. Previous point cloud generation works such as PVD and LION conducted experiments on multiple conditional generation tasks to verify their design. However, this paper only shows the results of unconditional generation. I believe more experiments such as point cloud completion or 2D image-to-3D point cloud should be included. 3. The qualitative results shown in the main paper only contain examples generated from DiT-3D and the supplementary material only additionally contains results from PVD, DPM, and SetVAE. As two methods that are quantitatively most comparable to DiT-3D, the visualization of LION and MeshDiffusion are not shown in this paper. So it is not clear how DiT-3D outperforms these methods. A minor issue: the computation resources should be detailed in this paper. [1] SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds (ECCV 2022)

Questions

Please refer to the weakness part

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper tackles the task of 3D generation. Inspired by the recent progress of utilizing transformers in 2D image generation with diffusion processes (DiT [1]), this paper proposes to replace the common U-Net in 3D diffusion models with a plain transformer. To adapt the DiT to the 3D scenario, authors make several modifications to the vanilla 2D DiT. Experiments on ShapeNet demonstrate improvements over baselines.

Strengths

This paper tackles an important task of 3D generations that is important for many downstream tasks. The paper is generally well-written and easy to follow. Experiments are thorough and convincing.

Weaknesses

My main concerns are about the incremental development of the proposed DiT-3D based on DiT [1]. Specifically, a. **Diffusion on voxelized point cloud** has been studied in [12]; b. **3D positional embeddings** is a natural/must-have modification from DiT's 2D positional encodings; c. **3D window attentions** may not be necessary. Actually, I am quite confused why we need the following procedure: point cloud -> voxel (Sec. 3.2 Voxelized point clouds) -> patches in voxels (Sec. 3.2 patch embeddings) -> reshape patches in voxels into 3D window (Sec. 3.2 3D window). I think these hierarchical steps essentially just change the **actual voxel size**. Then why don't we just have a voxel with a resolution of $(p \cdot R)^3$ at the very beginning? Here $p$ is the patch size the author used in "3D Positional and Patch Embeddings"(Sec. 3.2) and $R$ is the number of patches for "3D window" (Sec. 3.2). And this does not prevent authors from applying 3D convolution to exchange information (L179). If the above abstraction/simplification is correct, it seems like the major modifications to 3D are just changing from image patches to voxels. d. Authors state > These results indicate that our DiT-3D can support flexible transferability on modality and domain, which is different from previous 3D generation methods [12, 13] based on U-Net as the backbone of DDPMs (L325-327). Can the authors explain whether there are some experiments to support this statement? As in Tab. 3, we only have results from DiT-3D without any baselines.

Questions

1. Authors state > 3D shape generation is a challenging and important problem that seeks to synthesize high-fidelity point clouds ... (L27) I do not think this is a proper statement as there are many 3D representations besides point clouds, e.g., mesh [15] or implicit ones [A, B]. 2. L190 "Q, K, V have the same dimensions" where the notation "V" is the same as the voxel resolution in L176. Please modify it to make readers be able to distinguish the two. 3. L226 "It’s worth noting that we initialize $\gamma$ to 1, which is then multiplied with the frozen layers": $\gamma$ is not explained. 4. L266 "trained on point clouds (l-GAN) and latent variables (l-GAN)": duplicated 1-GAN. [A] 3D Neural Field Generation using Triplane Diffusion. CVPR 2023 [B] SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation. CVPR 2023.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No limitations are provided by the authors.

Summary

This paper proposes to adapt Diffusion Transformer [1] to class-conditional 3D point cloud generation task. In order to achieve it, the authors propose (1) to apply diffusion/infusion directly to point clouds rather than work in latent space; (2) transform input point clouds to voxel grids to apply transformers to tokens extracted from 3D grids in a straightforward way; (3) to reduce the complexity voxel features are processed in patches to produce patch tokens and self-attention in the transformer is modified to aggregate tokens in a window of a predefined size; (4) final voxel features are devoxelized back into points for per-point noise prediction in the infusion process. Experimental results show promising results in class-specific point cloud generation on ShapeNet dataset, various ablation studies demonstrating the importance of different components and some additional transferability studies that use selective fine-tuning to adapt models pretrained on a data modality/domain to another one.

Strengths

The main strength of the paper lies in experimental results beating state of the art with an novel approach, not based on prior 3D point cloud generation works. Although it is an adaptation of Diffusion Transformers working with images to 3D point clouds, such a transition from one data type to another is not trivial, so it is remarkable that the authors made it work in this setting.

Weaknesses

In my opinion the main weakness of the paper is the quality of the presentation. Text can still be polished to improve readability and correct some mistakes. The authors claim across the paper (e.g. L120) that they apply diffusion for the voxelized point clouds, and I think it is misleading. In fact the diffusion is applied to regular point clouds, it is the infusion network that is designed to operate on voxelized 3D features, but as far as I understand, the denoising is applied on a per-point basis. Clarity of explanation can also be improved, since some details are missing (see questions). The authors show a lot of ablation studies, but comparisons to external approaches are limited by the main single-class generation experiment. Qualitative comparisons are moved to supplementary materials, while it is better to show it in the main paper (quality of these comparisons could be improved by decreasing point sizes and sampling more points per shape, so some finer details could be examined).

Questions

1. In principle, point to voxel and backwards transitions do not necessarily preserve all the points, since if the resolution is low, several point will collapse into a single voxel and will not be recovered during devoxelization. How is this avoided in this work? 2. Since evaluations in this work only include single-class data setups, the expressivity of the proposed model is left underexamined. LION included some experiment showing that their approach is capable of generating realistic point clouds when pretrained on multi-class data. It would be interesting to compare how your class-conditioned model will perform in that setting. 3. What is a «plain transformer»? How this «plain» property is characterized? I think, it is better to drop this adjective or change it and be more specific. Minor comments: * The optimal best possible value of 1-NNA metric is 50%, since it means that the nearest neighbour classifier is incapable of distinguishing true from gerenated samples. The text and tables should be modified accordingly. * L14: «high computation» -> high computational costs * L119, L340: «operate» -> perform or implement * L234: sentence is broken

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not provide any statements about such limitations.

Summary

This paper proposes a Diffusion Transformer for 3D shape generation (point cloud), named DiT-3D, which conducts the denoising process on voxelized point clouds. Technically, it introduces 3D positional and patch embeddings, as well as 3D window attention. The main experiments are done on ShapeNet. In addition, the authors empirically show that the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet.

Strengths

- The paper is clearly written and easy to follow. - The authors extend the 2D window attention operator to 3D. - The authors empirically show the benefit of leveraging a 2D transformer pre-trained on natural images for 3D generation.

Weaknesses

1. It seems that an important baseline or reference is missing: "Point-E: A System for Generating 3D Point Clouds from Complex Prompts". In L172-L173, the authors claim that "We tried to train the diffusion transformer on point coordinates, but it did not work since point clouds are sparsely distributed in the 3D embedding space". However, given Point-E, it sounds not that convincing. Can the authors explain why Point-E is not mentioned or compared in the paper? 2. It is hard to tell whether the generated shape is of high fidelity if the number of points is only 2048 (L245). It is more convincing if the number of points is larger than 4096 (Point-E) or 16384 (usually used in high-fidelity point completion). In addition, it will be visually better if the authors can present the 3D shapes in the format of mesh (like Point-E, MeshDiffusion) or colored point clouds. Currently, it is hard to tell whether the point cloud is of high fidelity. Visually, GET3D and Point-E look better than this work. 3. Only ShapeNet is used. Currently, there are more and more 3D datasets. It will be better if the authors can show results on more categories of ShapeNet, ABO, or (a subset of) Objaverse.

Questions

Since the evaluation metric is based on Chamfer Distance or Earth Mover’s Distance, I assume that the authors sample a fixed number of points from the GT mesh (and predicted mesh if the baseline, e.g., GET3D, is a mesh-based), which is 2048. I am not sure whether the metric might favor the proposed method especially when the number of points is small, as details can be missed under such a condition. The authors can try to use a larger number of points to compare baselines with the proposed method.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitation is not adequately addressed.

Summary

The paper introduces DiT-3D, a groundbreaking diffusion transformer for 3D shape generation. It addresses the limitations of previous 3D diffusion methods that mainly relied on the U-Net architecture. DiT-3D leverages the power of Transformers to directly operate the denoising process on voxelized point clouds, resulting in superior scalability and high-quality generations. The authors incorporate 3D positional and patch embeddings to aggregate input from voxelized point clouds and mitigate the computational cost of self-attention by employing 3D window attention in Transformer blocks. The proposed DiT-3D achieves state-of-the-art performance on the ShapeNet dataset, showcasing its ability to generate diverse and high-fidelity 3D point clouds.

Strengths

1. DiT-3D achieves state-of-the-art performance in single-category point cloud generation, demonstrating its effectiveness in generating high-quality 3D shapes. 2. The utilization of pre-trained DiT-2D checkpoints from ImageNet to improve DiT-3D on ShapeNet showcases the transferability of 2D diffusion models to the 3D domain, which is an interesting and promising approach. 3. The model is concise, and the paper is well-written, accompanied by clear and visually appealing illustrations.

Weaknesses

1. The authors only conducted unconditional generation experiments for single-category and three-category cases, limiting the application fields. 2. Although the window attention technique is employed to mitigate computational costs, there are concerns regarding the generation speed, when operating on 32 * 32 * 32 (even 64 * 64 * 64) voxel grids. 3. Most DiT-3D models, except for DiT-3D-S, have parameters exceeding 100 million, considerably higher than other existing 3D generation methods.

Questions

1. Have the authors considered exploring diffusion in the latent space? This may potentially enhance the overall performance and inference speed. 2. It would be more valuable to investigate multi-category generation, such as training on entire ShapeNet-13, ShapeNet-55, or even larger datasets like Objaverse [Deitke et al., 2023]. Previous work [Sanghi et al., 2022, 2023] suggests that voxel representation could aid in generalization to some extent. [Deitke et al., 2023] Objaverse: A universe of annotated 3d objects. In CVPR. [Sanghi et al., 2022] CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation. In CVPR. [Sanghi et al., 2023] CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language. In CVPR.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N.A

Following comment

Thank the authors for the reply. Although I still think the novelty is limited, I appreciate the effort of conducting new experiments. Therefore, I will raise my rating to 4.

Response to the authors

I am grateful for the explanations provided by the authors, which address my concerns to some extent. I'll keep my positive rating.

Additional response to the reviewer

Dear Reviewer trpZ, Thank you for your detailed review and the valuable feedback. We have carefully addressed each of your concerns and provided clarifications in our previous response. We would like to kindly request your response to the provided explanations and revisions. We appreciate your thorough evaluation of our work, and your feedback will greatly contribute to the improvement of our manuscript. Thank you for your continued engagement and support.

Rebuttal Reply

First, I appreciate authors's time and effort in addressing my concerns. Regarding comparing to PVD: I am not sure whether I fully get authors's arguments. Can authors clarify what it means for "we are the first to use a plain diffusion on voxelized point clouds". I think PVD is also a plain diffusion process. Regarding 3D positional encoding: I still think the novelty is quite limited for claiming 3D positional encoding as a major contribution (L73) but I appreciate authors's effort in showing that this is an important factor for achieving high-quality results (Tab. 2). Regarding 3D window attention, I think my concerns get resolved. However, I am still confused about the process of the series of voxelization and patchification descried in Sec. 3.2. Essentially, point clouds first get voxelized into $V^3$. Then without going through any networks, it will get reshaped into $(V/p)^3$ and fed into the first network (L179). I think essentially, we only have a voxel with a resolution of $(V/p)^3$. In essence, assume the original voxel size is $u$. Why not directly have voxelization for voxel size $u \cdot p$ at the very beginning? Meanwhile, after reading other reviews and authors's responses, I think this work provides some benefits to the community. However, I am still quite concerned about the limited novelty of the techniques used in the paper. Based on this, I raised my score to 5.