GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

Summary

The paper introduces a novel explicit and structured 3D representation - GaussianCube - used in combination with U-Net diffusion models for 3D generation. While 3D Gaussians have a lot of advantages, they are not directly compatible with efficient architectures for generative modeling because of their unstructured nature. This paper tackles this problem by proposing two steps from 3D Gaussians to GaussianCube: 1) densification-constrained fitting to bound the number of Gaussians, and 2) structuralization via optimal transport to convert the underlying unstructured point cloud of 3D Gaussians to a structured voxel grid. Using datasets in this format, the authors train diffusion models parameterized as 3D U-Nets for object, avatar, and text-to-3D generation.

Strengths

* The paper introduces a simple but effective idea of bringing structure into unstructured 3D Gaussians. - This small change to the representation makes it directly compatible with standard architectures for diffusion models. - The use of optimal transport for voxelization is novel and elegant. - The representation might have potential to enable more efficient / optimized network architectures in other 3D tasks as well. * The paper is presented well: - The main approach is clear. - The paper is well-written and easy to follow. * The paper provides an extensive experimental evaluation validating their approach: - Generation with GaussianCube is evaluated for 4 tasks on synthetic as well as real data: unconditional generation on ShapeNet, class-conditional generation on OmniObject3D, image-conditioned avatar generation on Synthetic Avatar, and text-to-3D generation on Objaverse. - Quantitative and qualitative results show strong performance compared to baselines. - The supplementary material includes a video with more convincing qualitative results. - The ablation study validates the design choices.

Weaknesses

* Lack of clarity: - The paper does not include any details about the diffusion process on the properties of 3D Gaussians (except of normalization with dataset statistics in the appendix): - 3D Gaussians have parameters with very different distributions making the diffusion process on them non-trivial. - As one example, if rotations are represented as quaternions with a diffusion process adding simply Gaussian noise, there is no guarantee about the final generation being a valid rotation. - Also the distribution of offsets from voxel centers would be insightful, especially across different objects/scenes with different level of detail. - The fact that positions are replaced by offsets of the voxel centers became clear rather late in the paper (ll. 153 ff.) and should be explained earlier. - The optimal transport illustration in Fig. 3 is not intuitive. Are there only two valid pixel corners on the diagonal? * Missing related work and baseline: - The related work about diffusion models on 3D representations is missing [1] and [2]. - The state-of-the-art comparison on ShapeNet misses an important baseline [1] with strong performance. [1] Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction, ICCV 2023 [2] Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation, CVPR 2024

Questions

* Why do you use $x_0$-parameterization instead of $\epsilon$ or v? * The resolution 32 is rather small and the appendix shows stronger fitting performance for larger numbers of Gaussians. Is there potential for latent diffusion? * Why is the reported performance of DiffTF worse than the one from the original paper?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed limitations and broader (including potential negative societal) impact in the appendix.

Summary

The paper is about 3d object generative model. The generative objects are represented with gaussian splatting. Thus the main obstacle is the large point clouds. It is difficult to generate such large point clouds. The authors proposed a way to put point cloud to a structured grid using optimal transport. Some generative experiments are shown.

Strengths

The idea of using optimal transport here is creative. And there is a motivation behind this. The figures are also nice to illustrate the idea. There are some kinds of experiments to verify the idea, including, objects and human avatars. Generally I like the proposed idea and the convincing results shown in the main paper (and the video).

Weaknesses

I would like to see some time and memory analysis about the method, including data preprocessing (object fitting and optimal transport), training resources, and sampling speed. These are important for readers to understand how difficult it is to reimplement the method. Another thing might be useful is novelty analysis. Shapenet is a relatively small dataset, which can be easily memorized.

Questions

The digital avatar experiment is image-conditioned generation? Is it possible to train an unconditional generative model? I do not have further questions for this paper.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper proposes an approach to 3D generation at the object level with the help of diffusion models. The main challenge in using diffusion model to generate 3D is the choice of 3D representation that fits well with the denoising network. The paper proposes using fixed number of Gaussians lying on a regular 3D grid. Each cell of the 3D grid stores the parameters of the Gaussians along with an offset from the cell. However, the Gaussians trained using Gaussians Splatting can lie on arbitrary coordinates. In order to assign a Gaussian to the cell of the grid, the paper proposes using Optimal Transport algorithm. This datastructure is called GaussianCube and result in shape being represented with smaller number of parameters. Based on this GaussianCube data structure, we can train a 3D diffusion model to generate 3D shapes via Gaussians. This network can also be used to predict Gaussians in conditional manner e.g. image, text etc. The resultant generative model produces 3D consistent representation via Gaussian splatting, follows conditioning signal accurately, fast inference (5 sec) and lower number of parameters.

Strengths

The paper proposes a novel approach to consistent 3D generation with the help of diffusion model. The formulation of 3D dataset structure GaussianCubes is interesting as it allows 3D convolution on an unstructured data i.e. set of Gaussians. The paper is mostly well written and describes the proposed approach well. Experiments are performed on relevant dataset and evaulated against valid baselines. Overall it is a good work showcasing how to combine diffusion model with Gaussian splatting in an end to end training resulting in better performance across several benchmarks.

Weaknesses

1. Despite good results, my main concern is in the scalability of this approach. Given enough compute, we will be limited by the amount of 3D data. This is one of the reason of many efforts in 3D generative models based on images or videos only, as they are in abundance. I am skeptical can be used to generate assets for real applications. (Authors rebutting this should focus on how data scalability be achieved?) 2. How expressive and generalizable is the model in term of fitting many datasets (shapenet, objaverse etc) jointly? (Authors should not do experiments if they have not already tried training on all datasets.) 3. I find it quite surprising that the GaussianCube data structure works well even when trying to fit shapes with different scales and dimension. I expect that this representation will be hard to scale for complicated topology and intricate details where number of gaussians required to approximate the shape will be quite large. (Authors should discuss it in the rebuttal.) 4. Missing training time and compute requirement for training diffusion model. (Authors should provide this data in the paper.)

Questions

1. In my opinion, Figure 3 is not required. The textual explanation is enough. 2. What are the failure cases of optimal transport? Have authors considered a learnable scattering of gaussians to cubic grid? (Authors should discuss this in the rebuttal.)

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations and societal impacts are described in the Supplementary material. Though the scope of the paper is 3D asset generation and experiments are done accordingly, I would have appreciated some discussion on large scale scene generation also. I suspect that there are failure cases of optimal transport. If so, they should be described.

Summary

The paper proposes a new structured and explicit representation based on 3D Gaussian Splatting for 3D generation. The key idea is to properly organize the 3D Gaussians into a fixed-size volume, allowing for the use of the standard 3D UNet for diffusion. First, the paper uses a densification-constrained fitting algorithm to constrain the number of 3D Gaussians to be no more than a fixed value. Then, the paper proposes to use Optimal Transport to obtain a better spatial structure for the 3D Gaussians. The proposed representation can be easily used for both unconditional and conditional 3D generation. Experiments show that the proposed method outperforms baseline approaches.

Strengths

Originality: The paper proposes a new method to encode 3D Gaussians into a structured and explicit voxel grid, which is novel. The key novelty differs from the concurrent work [21]. While [21] also uses a voxel grid to organize 3D Gaussians, this paper proposes using Optimal Transport to make learning easier. This might inspire future research. Quality: The paper is technically sound. The author provides detailed implementation details. Clarity: The paper is mostly clear and easy to read. Significance: Applying diffusion models to 3D Gaussians for 3D generation is an important research problem, and the paper provides one feasible solution to this problem. This problem is non-trivial due to the irregularity of 3D Gaussians. The paper directly addresses this issue and proposes using OT for better spatial organization. Overall, I think this paper might inspire future research.

Weaknesses

There are some concerns that need to be clarified: - In Table 2, how is the number of the Triplane parameters computed? Why is it larger than Voxels? In literature, people use Triplane because it is more efficient than voxels. In addition, L11-L13: when encoding a large dataset, do Triplane/voxels (with shared implicit decoder) still have more parameters than GaussianCube? - Is Equation (4) used to train the diffusion model? How do you obtain I_pred? (Do you run multiple denoising steps to generate it?) How does the expectation depend on t here?

Questions

- The ShapeNet dataset is axis-aligned, but Objaverse is not. Objects in the Objaverse dataset might have different orientations. Is the method still able to generate meaningful results when the dataset size of Objaverse is small (e.g., only using ~5k cars from Objaverse)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A