3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.
Paper
Similar papers
Peer review
Summary
This paper presents a method to turn the representation of the gaussian splatting as a point cloud into a VAE representation. With that representation it is shown how to perform several tasks such as unconditional generation, conditional generation, and point-to-gaussian generation, all of which are done using a diffusion model that operates on the latent . It is shown that it performed better than previous methods on these tasks.
Strengths
The method presented in the paper is a novel way to incorporate the gaussian splatting attributes into generation tasks. The method shows better results on some tasks than previous methods.
Weaknesses
The main one is that while reading the paper I did not get the answer to the question why enforcing a VAE to predict gaussian splatting should work better than a simpler method. It was shown that this is the case by experiments, but what is the motivation? As SOTA NeRF methods produce better reconstruction than the vanilla GS. Since the paper presents a method to create a new 3d object from the gaussian splatting primitives the relevant papers from the early (1990-2002) use of gaussian splatting should be referred to. Additionally the number of quantitative results is limited (even in comparison to the paper of the referred baselines), and makes it hard to evaluate the method. Moreover, the second part of the first sentence in the abstract “... yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructural nature.” is IMHO logically challenging, only after reading the introduction it becomes clear that the authors referred to generation novel 3d assets using gaussian splatting. The main concepts and ideas in the paper are interesting, but there is still work required in both writeup and experimental parts.
Questions
see weaknesses.
Rating
4
Confidence
4
Soundness
2
Presentation
1
Contribution
2
Limitations
see weaknesses.
Summary
This paper proposes DiffGS, a novel diffusion-based generative model that can generate 3D Gaussian Splatting (3DGS) representing an object. As 3DGS is naturally discrete and unstructured, it is not practical to directly diffuse the 3DGS representation. Thus, the authors propose to reconstruct the 3DGS with structured triplanes that represent three functions: a probability function that models the geometry (the mean of Gaussians) of 3DGS, a color function that models the color, and a transform function that models the transformation parameters of the Gaussians. Additionally, the authors propose a VAE that encodes 3DGS into a latent vector. By representing 3DGS with such structured data, it is then possible to train a diffusion model on the latent vectors, which can be recovered to 3DGS for rendering. Experiments show that the proposed DiffGS achieves preferable quality results compared to baselines and by adding conditions to the diffusion model, it can achieve various applications.
Strengths
1. The paper is well-written and easy to follow. 2. The idea of representing 3DGS with disentangled functions is novel and practical. Indeed, it is difficult to diffuse unstructured data like 3DGS, and representing 3DGS using functions that can both encode and decode is a novel direction.
Weaknesses
1. It can be observed that there might be holes and incomplete shapes in the generated 3DGS (Fig.5(b)). I suspect the reason for this is that the proposed Gaussian extraction algorithm struggles to sample points in high-frequency parts like the legs of chairs. While this can be improved by increasing the number of sampled Gaussians or the depth of the octree, it might be quite computationally intensive. 2. The inference efficiency is not shown in the experiments. During each sample, DiffGS requires optimization over the sampling points, which may further slow down the inference speed of the model. The authors should provide the influence of the optimization on the inference time. However, it is acceptable if the inference speed is slower than baselines since diffusion models are naturally slow to inference. 3. An important baseline is missing. Splatter Image [1] is also a 3DGS-based object reconstruction method and shares the similar idea of representing unstructured 3DGS with structured data. Since the Splatter Image models trained on ShapeNet are also available, it is reasonable to compare DiffGS with Splatter Image under the single-view reconstruction task. 4. The authors do not provide the number of parameters of DiffGS and the comparisons with baselines. In the quality comparisons, the authors should keep DiffGS and other baselines having around the same number of parameters. [1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction, CVPR 2024
Questions
1. What is the inference time of DiffGS, and how much does the optimization process during inference influence the inference speed? 2. How many parameters does the model use?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes, the limitations are discussed in the paper.
Summary
The paper is about generating 3D objects by using gaussian splatting. The method converts the points (with gaussian properties) to continuous fields. With this idea, the irregular structured data can be easily processed by neural networks.
Strengths
The idea is interesting and novel. Usually we need many points (10k-1M) to represent a single object. With this large point clouds, it is very challenging to process the data with neural networks. Thus the paper proposed to convert a point cloud to a field which can be easily processed by neural networks. The octree design seems to be also interesting. This is useful to get fine-grained structure.
Weaknesses
The results shown in the supplementary video is not very satisfying. Even the paper showed some metrics comparing to some existing methods, the visual quality is much worse. Specifically, the metrics shown in Table 1 are better than some prior works. However, I believe some important works are missing in the table, e.g., EG3D, DiffRF, SSDNerf. The evaluation protocol seems to be different from these works. I am curious how the authors pick 50k images as the reference set. SSDNerf used testing set as the reference set. I know there is always a debate between training and testing set in the evolution of generative models. But this should be mentioned in the paper. For the visual quality, results of EG3D and DiffRF are much better than this paper. It would be better the authors can do a visual comparison with these works.
Questions
How do the authors optimize the loss in Eq 6? I understand how the point are converted to fields thus we can query the probabilities continuously in the space But how about other properties? They are not converted to continuous fields.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
3
Limitations
Yes
Summary
This paper proposes a new generative model for Gaussian primitive generation based on latent diffusion models. In detail, the method disentangles Gaussian Splatting generation into three functions, i.e., Gaussian probabilities, colors, and transforms. The method can achieve unconditional and conditional generation and extract Gaussians at arbitrary numbers via octree-guided sampling. Experiments on unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation, show the advances of the proposed method.
Strengths
1. DiffGS effectively deals with the discreteness of Gaussian Splats, by disentangling Gaussian Splatting into Gaussian probabilities, Gaussian colors, and Gaussian transforms. 2. DiffGS shows good performances on multiple tasks, including unconditional/conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. 3. DiffGS is able to generate high-quality Gaussian primitives at arbitrary numbers. 4. The paper is well-written and easy to follow.
Weaknesses
1. What is the benefit of predicting features in structural triplanes first, then extracting 3D Gaussian Splatting, compared to directly modeling the 3D using triplanes? The generation speed is not an advantage, and the performances are not compared/ablated in the experiments. 2. Are Gaussian probabilities work better than directly predicting point cloud explicitly? To show the advantage of the GS disentanglement proposed in this paper, it would be better to compare it with previous work, e.g., [1]. 3. Experiments can be strengthened by comparing with optimization-based methods for generating GS, e.g., [2], and GS prediction from multi-view Diffusion Models, e.g., [3]. 4. Mathematics. * $i$ and $j$. In lines 131-132, why "$j=1$" and "$j$-th" are used for $g_i$? Similar for the query points definition. * Eq. 3 typo. * Does bigger $\psi_{pf}$ mean larger probability? If so, do we want to maximize Eq. 9? Please double-check the math and add the necessary explanations. --- References: [1] Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. CVPR 2024. [2] DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. ICLR 2024. [3] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.
Questions
Please refer to the weaknesses section for more details.
Rating
5
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
Yes
Dear reviewer tnRu, Thank you for your response and the helpful comments. Following your suggestions, we conduct an additional ablation study under the same experimental settings outlined in Figure C of the rebuttal PDF. Specifically, we directly model the Gaussian attributes (e.g. color, density) using triplane and evaluate its performance in terms of PSNR, SSIM, LPIPS metrics, and inference time. The results are presented in Table E below. As shown, while the approach that directly utilizes triplane for Gaussian modeling without feature predicting and decoding slightly improves the inference speed, it leads to significant degradation on the quality of 3D Gaussians. In contrast, our proposed method demonstrates superior performance in terms of PSNR, SSIM, and LPIPS, compared to this alternative design. The results demonstrate the effectiveness of our triplane framework designs and the key insight to disentangle Gaussian splatting through three novel functions. **_Table E: Ablation studies on the framework designs of the triplane._** | | PSNR | SSIM | LPIPS | Inference Time (s)| | :-----------: | :-------: | :--------: | :--------: | :------------: | | Only Triplane | 23.55 | 0.9732 | 0.0340 | **9.1** | | Ours | **34.01** | **0.9879** | **0.0149** | 9.5 | Best regards, Authors
Thank the authors for the detailed reply. For the "Only Triplane" ("directly model the Gaussian attributes (e.g. color, density) using triplane") experiment, is there a NeRF decoder used after the triplane, like in EG3D and SSDNeRF? If not, would it be possible to also compare with this setting?
Response to Reviewer tnRu
Dear Reviewer tnRu, Thanks for your response and the positive assessment on our rebuttal. We appriciate your suggestions on the potential comparison with the use of a NeRF decoder (e.g. EG3D, SSDNeRF). In response, we conduct additional experiments where we replace our Gaussian functions with the SSDNeRF decoder to predict attributes of colors and density, maintaining all other settings constant. The results of this experiment are represented in Table F below. As shown, the SSDNeRF decoder results in a noticeable decline in PSNR, SSIM, and LPIPS metrics. The results demonstrate that our proposed Gaussian functions contribute significantly to the 3D rendering quality, showcasing the effectiveness of our functional decomposition of Gaussian representation and designs on the triplane framework. **_Table F: Ablation studies on the design of the decoder._** | | PSNR | SSIM | LPIPS | | :-------------: | :-------: | :--------: | :--------: | | SSDNeRF Decoder | 28.74 | 0.9838 | 0.0218 | | Ours | **34.01** | **0.9879** | **0.0149** | We further demonstrate that Gaussian splatting provides a more efficient solution for high-fidelity 3D rendering compared to volume-based methods like NeRF. By adopting a point-based representation for optimization and rendering, it reduces computational demands and accelerates the convergence of the training process. Most importantly, 3D Gaussian splatting significantly enhances rendering speed, making it particularly well-suited for real-time applications such as virtual reality and interactive 3D simulations. In practice, the rendering time for each frame using 3D Gaussian splatting can be less than 0.01 seconds, compared to several seconds for NeRF. We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further. Best regards, Authors
Summary
This article first transforms 3DGS into a regular representation: the triplane structure, to separately model the location probability, color, and other attributes of each Gaussian. This representation can then be used to train a generative model through a VAE and LDM. To sample Gaussian positions from the triplane structure, this work uses an octree to partition the space based on the magnitude of the location probability. It initializes the Gaussian point positions by sampling from the nodes with higher probability, and then further optimizes the final positions.
Strengths
Strength: 1) This paper is well written and easy to understand 2) This design supports versatile downstream tasks: test/image conditional GS generation; GS completion; Point-to-GS Generation
Weaknesses
Weakness: 1) This work is only trained on a small dataset. Similar works such as "triplane meets gaussian splatting" are trained on the Objverse dataset, which can visualize more diverse results. This might be due to the cumbersome data processing, and will limit the scaling up in the future. 2) The equation (3) is a little weird. Why the choice of \tau matters as shown in the ablation study? 3) The 3DGS representation is not properly evaluated. Given an existing 3DGS, is it possible to be reconstructed by firstly represented as GauPF, GauCF, GauTF and then recover by Octree-Guided geometry sampling? 4) Personally, I'm not sure whether 3DGS is a suitable representation for 3D generation since it's highly irregular, which cannot be embedded into a generative model conveniently. Moreover, the data processing is really time-consuming, which will further limit the large-scale training. 5) More comparison with GS-based reconstruction model should be added, such as "LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation"
Questions
1) It seems the optimization target in equation (9) is wrong. Should it be -log(\psi(\rho))
Rating
4
Confidence
5
Soundness
3
Presentation
2
Contribution
2
Limitations
The author has shown some failure cases and limitation of this papar.
Summary
The paper introduces DiffGS, a text/image-to-3D generative model with 3D Gaussian splatting as its output representation. The model generates Gaussians from text/image condition by CLIP-augmented latent diffusion model (LDM), whose output is a latent vector that can be decoded into a triplane representation. DiffGS then translates this triplane representation by first sampling a plausible set of Gaussian locations from the learned implicit function (GauPF), and then reconstructs colors and shapes of these Gaussians with two more learned implicit functions (GauCF, GauTF) which are used as components of the objective function in the optimization-based generation. Therefore, the model consists of a (1) Gaussian LDM, a (2) latent-to-triplane decoder and (3) three implicit functions (GauPF, GauCF, GauTF). The (2, 3) decoder models (triplane decoer and three implicit functions) are trained in a VAE fashion, and the (1) Gaussian LDM is trained over the top of VAE’s learned latent space. The method is tested with ShapeNet and DeepFasion3D dataset with FID/KID metrics.
Strengths
1. The authors have attached supplementary materials that effectively show their code and outputs of their method. 2. The paper contains ablation study (Table 2) that justifies each component of their framework. 3. Visual results show that the proposed method effectively generates complicated geometries with Gaussian splatting representation.
Weaknesses
1. Since the paper targets text/image-to-3D generation with Gaussian splatting based on the domain-specific (e.g., ShapeNet/DeepFasion3D) latent diffusion model, I believe that the results should be compared with other similar 3D generative models, at least those mentioned in Section 2.2 by the author. **Particularly, the work seems to solve similar problem with LGM (ECCV 2024, code released), and therefore should be thoroughly compared with it.** 2. **The detailed structure of the proposed models are not presented in the paper.** For example, that is the dimension of the latent z in Figure 2, and why did the authors have chosen the structure? How does the authors model the Gaussian Splatting encoder that encodes the (fixed or not fixed number of) Gaussians into this latent z? Does the number of Gaussians fixed in this model? How are the architectures of GauPF/GauCF/GauTF designed? How does the DALLE-2’s LDM, which is originally proposed to model the latents of a 2D image, be used to model the latent triplane representation? How does the triplane decoder shaped? How many parameters are used in the models? Such information should be included in the main manuscript (or at least in the appendix) for proper presentation of the idea. 3. The proposed method uses post-optimization steps to infer the generated 3D Gaussian splatting. This seem to require **additional generation time in the inference steps.** This can be considered a weakness since there are already many large generative models of Gaussian splatting that generate samples in tens of seconds such as LGM. Even if the proposed model requires long delay, such delay can still be justified if the results are sufficiently detailed and the sampling time is sufficiently short (as how we acknowledge the contribution of 2D LDM models). However, there is no information of the exact generation time specified in the paper. 4. The positive correlation between the number of Gaussians / octree depths and the reconstruction quality enhancement in Table 3 and 4 seems to be trivial. I believe more interesting question is **how much the amount of increment in computational cost trades off against the quality enhancement**, etc. 5. Since the model uses CLIP to perform text/image-to-Gaussian generation, the paper should report **numbers that show the text-fidelity, e.g., CLIP-scores**. Comparing only with FID score (Table 1) does not seem to effectively demonstrate the superiority of the proposed method. As a summary, my key concerns are the presentation of the materials, the lack of comparison with the more recent papers (e.g., LGM) that appears in Section 2.2 of the main manuscript, justification of the sampling-time delays that seem to present, and the lack of quantitative comparison on text-fidelity. Unless these concerns are resolved, I believe that the paper is not ready to be published in the venue.
Questions
1. Although the authors claim for the generation of 3D Gaussian splatting, the actual generated sample is the triplane representation, which is realized as a Gaussian splatting with implicit neural representation. Since there are already a well-established triplane-to-NeRF decoding (e.g., TriNeRFLet: A Wavelet Based Triplane NeRF Representation, ECCV 2024, code released), are there **specific technical benefits of decoding this information into Gaussian splatting in this case?** 2. The proposed work involves a dedicated generative model specific to certain domains (e.g., ShapeNet). However, there are other branches of works that approaches text/image-to-3D generation leveraging the power of 2D diffusion models, e.g., DreamFusion and its offsprings that uses SDS losses, which seem to be mature at this moment. Concerning the recent achievements of the latter branch of works, I am not fully convinced to training a domain-dedicated 3D generator. May I ask **why the authors think training the dedicated 3D generator is a good way of solving 3D generation tasks?** 3. In line 109: lake → lack Please note that these questions are not counted in my overall scoring.
Rating
5
Confidence
5
Soundness
2
Presentation
1
Contribution
2
Limitations
Yes, the limitations are addressed in the appendix.
Thank you for your efforts to address the concerns. Here is one point that needs further clarification: as mentioned in Weakness 1, an ablation study comparing the proposed method and the one that only uses triplane to predict color and density would be beneficial to show the advantage of the proposed approach.
Looking forward to the discussion
Dear Reviewers, We sincerely appreciate your comments and expertise. Please let us know if there is anything we can clarify further. We would be happy to take this opportunity to discuss with you. Thanks, The Authors
Dear Reviewers, As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification. Thanks, The Authors
I appreciate the authors for their careful and detailed rebuttal with additional experiments that have resolved many of my initial concerns. I have a few remaining questions regarding the rebuttal. --- **Global-Q1: Evaluation Against Other SOTA 3D Generative Models** Regarding that the domain-specific SotA methods compared in the additional experiments such as LGM are trained with Objaverse dataset, it might be unfair to directly compare this with ShapeNet-fitted DiffGS for the generation of ShapeNet data. Would you share your thoughts on how DiffGS can be better (in terms of diversity and quality) if it is scaled to Objaverse dataset based on the shown experiments? --- **Q7: Justification for Training a Domain-Dedicated 3D Generator** According to Appendix C of the manuscript, I see training for ShapeNet already takes a week for a 8-GPU server. I think in order to make the contribution of this work practically useful, the method should be scaled up to at least Objaverse-scale datasets (perhaps in the future. I do not believe it is adequate to mandate conference papers to train on huge datasets, so I am not requiring the authors to do the experiments.) I am a little bit suspicious on the computational feasibility of this scaling up process. For example, training a 100-GPU cluster for six month for getting a domain-specific Gaussian splatting generator would be a waste of time compared to having a 10-minutes image-to-3DGS generator such as LucidDreamer [1]. Would you persuade me that the proposed approach is computationally feasible and practically meaningful? --- These are the last of my concerns. [1] Yixun Liang, et al., LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching, CVPR 2024.
Response to Reviewer p5pi (1/2)
Dear Review p5pi, Thanks for your response and the positive assessment on our rebuttal. We response to each of your additional questions below. Please do not hesitate to let us know if you have any additional questions. **Discussion-Q1:Thoughts on the superiorty of DiffGS in terms of diversity and quality when scaling up** We believe that DiffGS offers a more flexible and effective approach to 3D Gaussian representation and generation. By disentangling the 3D Gaussian Splattings (3DGS) into three novel functions to model Gaussian probabilities, colors, and transformations, DiffGS delivers significant advantages in Gaussian quality, generation efficiency, and generation diversity. 1) DiffGS can scalably generate Gaussian primitives at arbitrary numbers. All the previous works that explicitly reconstruct Gaussians can only generate limited numbers of Gaussians. For example, TriplaneMeetsGaussian generates up to 16,384 Gaussians, and LGM generates up to 65,536 Gaussians. In contrast, we novelly introduce the GauPF, which models Gaussian probabilities using neural functions. A specially designed discretization algorithm to used to extract Gaussian geometries from GauPF by octree-guided sampling, enabling generating Gaussians at arbitrary numbers. The number of Gaussians is a critical factor in the rendering quality of 3D Gaussian Splattings. As shown in the ablation studies on Table 3, the number of Gaussians significantly affect the quality of Gaussian reconstructions. With the ability of generating arbitrary numbers of Gaussians, DiffGS offers a naturally superior solution for 3D Gaussian generation. 2) We designed DiffGS using latent diffusion models (LDM), which significantly enhances both the efficiency and generation diversity of DiffGS. While most popular approaches for creating 3D Gaussian Splatting (3DGS), such as LGM and TriplaneMeetGaussian, primarily focus on "reconstructing" 3DGS, following the recent trend of large reconstruction models (LRM), they fall short in "generating" 3DGS. The reconstruction-based methods do achieve convincing results on the learned data domain, but their capability of creating diversity 3D shapes are limited due to the "reconstruction" target. In contrast, diffusion models have demonstrated a strong capability in generating diverse samples, both with and without conditions. Leveraging diffusion models, DiffGS shows great capability in generating diverse and high-quality 3D Gaussians compared to the "reconstruction"-based method LGM. The comparisons are shown in Figure A and Figure B of the rebuttal PDF. Moreover, by training DiffGS with a diffusion model at latent space (i.e. LDM), it achieves remarkable efficiency in generating 3D Gaussians. As shown in Table B of the rebuttal PDF, DiffGS outperforms LGM, SSDNeRF, DiffTF, and Shap-E in terms of speed, even when generating 350K Gaussians, which is significantly more than the number of Gaussian that previous works (e.g. LGM) can produce. 3) DiffGS is capable of generating 3D Gaussians unconditionally or conditionally from text and images. Moreover, DiffGS is the first model capable of solving the task of Gaussian completion and Point-to-Gaussian generation. In contrast, previous models like TriplaneMeetGaussian, DreamGaussian and LGM can only generate Gaussians from images or text. Though the extensive experiments and applications, we demonstrate DiffGS is a general and flexible framework suitable for most of the tasks taking Gaussians as the generation target.
Response to Reviewer p5pi (2/2)
**Discussion-Q2:Feasible and practical meaning of DiffGS compared to SDS-based approaches** We believe that training a native 3D generator like DiffGS offers a more promising future for 3D content creation compared to SDS-based methods. **(1)** SDS-based approaches require time-consuming optimization to distill 3D geometry and appearance from 2D diffusion models, often taking hours to converge. State-of-the-art methods like DreamFusion, Magic3D, MVDream, and Rich-Dreamer require 2-5 hours on one A100 GPU for optimization, as reported in their papers. The recent work LucidDreamer converges faster but still requires 36 minutes with one A100 GPU as reported in its paper. In contrast, DiffGS takes only less than 10 seconds for inference on a single 3090 GPU, which is several orders of magnitude faster than SDS-based methods. We further justify that training DiffGS for one day with eight 3090 GPUs is sufficient to achieve convergence for each class in ShapeNet. The time (5 days) listed in the Appendix C is for the full convergence, which is not necessary. We report the generation performance of training DiffGS for 12 hours, 24 hours and 5 days in Table G below. As shown, with the utilization of LDM, training DiffGS for one day is enough for convergence. It's important to note that DiffGS was trained on 3090 GPUs, which are significantly slower and have much less memory compared to the commonly used A100 GPUs (24GB vs. 80GB). The results highlight DiffGS's ability to scale to large datasets. We estimate that training DiffGS on the large 3D dataset Objaverse, which contains 1 M shapes, will take approximately 5-7 days to converge under 64 A100-80G GPUs. I believe the time is acceptable since we only need to train DiffGS for once. **_Table G: Ablation studies on the training time._** | | FID-50K Chair | KID-50K(%）Chair | FID-50K Airplane | KID-50K(%）Airplane | | :------: | :-----------: | :--------------: | :--------------: | :-----------------: | | 12 hours | 42.81 | 3.773 | 57.16 | 5.793 | | 24 hours | 36.69 | 2.244 | 50.04 | 3.864 | | 5 days | **35.28** | **2.148** | **47.03** | **3.436** Due to the limited time and computing resources during the discussion period, we are not able to train DiffGS under the large-scale Objaverse dataset. We will conduct experiments that scale up data sizes in the revised version. **(2)** SDS-based methods often lead to inconsistent generations, particularly the multi-head Janus problem. The 2D image diffusion models used in SDS lack an explicit understanding of both geometry and viewpoint. This absence of perspective information and explicit 3D supervision can result in the multi-head Janus problem, where realistic 3D renderings fail to achieve view consistency, causing every rendered view to be perceived as the front view. Although some works attempt to address this issue by incorporating view information into 2D image diffusion models (e.g., Zero-1-to-3, MVDream), they require both time-consuming 2D diffusion training (several weeks) and per-shape optimization. In contrast, native 3D generators like DiffGS are directly trained on 3D shapes, inherently ensuring 3D consistency and naturally avoiding the Janus problem. **(3)** A geometry initialization from native 3D generators significantly benefits SDS-based methods. The experimental results of recent SOTA SDS-based methods also demonstrate the necessity of introducing native 3D generation priors as the geometry initialization. We take LucidDreamer, which is mentioned by the reviewer, as an example. It leverages the generation results of Point-E, which is a native 3D generator taking colored points as the generation target, as the geometry initialization for SDS-based optimization. The ablation study shown in Figure 7 of the LucidDreamer paper demonstrates that the generation performance degrades without using the Point-E generations as the initialization for introducing consistent-aware 3D information. We believe that the advancements in the native generator 3D (e.g. DiffGS) will further contribute to the improvements of SDS-based methods (e.g. LucidDreamer) by providing superior geometry initialization. We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further. Best regards, Authors
Thank you for providing additional information. I will elevate my score as most of my concerns are resolved. I will be looking forward to large, practical application of the method. My final comment is a gentle recommendation for a revision of the manuscript based on the discussions in this rebuttal period.
Thanks to Reviewer p5pi
Dear Reviewer p5pi, Thank you very much for your insightful comments and positive assessment. We will certainly include all the new experimental results, along with the insights and analyses discussed during the rebuttal period, in the revised version of the manuscript. Following your suggestions, we will explore training a large and practical DiffGS model using the Objaverse dataset and will incorporate those results in the revision. We really appreciate you for the time you dedicated to evaluating our work and for upgrading the score. Best regards, Authors
Thanks for the answers. We'll keep the rating.
More efforts and motivation is required.
reply
Overall, I am positive about the methodology. My concern is about the results. The additional results of EG3D shown in the rebuttal seem to be much worse than the original paper. It is still unclear how to build the continuous version of other Gaussian properties (colors, rotations, etc) from the context. It looks like the only way is to use spatial interpolation with some kind of spatial kernels like RBFs.
Response to Reviewer 6WmW (1/2)
Dear Reviewer 6WmW, Thanks for your response and the positive assessment on our methodology. We response to each of your additional questions below. Please let us know if there is anything we can clarify further. **Discussion-Q1:The performance of EG3D** The visualization results of EG3D, as shown in Fig.D of the rebuttal PDF, were directly generated using the official code and pretrained models provided by its authors. For a fair comparison across all methods, we present the randomly selected samples of EG3D in Fig.D of the rebuttal PDF. We would like to justify that the performance of EG3D is not consistently as good as depicted in their paper, where only a few visually appealing generated cars are showcased. We also refer the reviewer to the visualization results of EG3D in Fig.5 of the DiffTF paper, where EG3D similarly produces results that are much worse than those reported in the original EG3D paper. The third-party reproduction by the DiffTF authors aligns closely with our reproduction of EG3D. Furthermore, the quantitative comparisons in Table 2 of the DiffTF paper clearly demonstrate that GET3D and DiffTF outperform EG3D in terms of generation quality, while DiffGS significantly surpasses both GET3D and DiffTF, as demonstrated by all the comparisons in Sec.4 and Sec.E of the Appendix. To further highlight the superior performance of DiffGS, we conducted extensive experiments during the rebuttal period, comparing our method against several state-of-the-art baselines, including EG3D (CVPR 2022), SSDNeRF (ICCV 2023), Shap-E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM (ECCV 2024), and DreamGaussian (ICLR 2024). The results demonstrate that DiffGS achieves visual-appealing generation results and outperforms all the SOTA baselines.
Response to Reviewer 6WmW (2/2)
**Discussion-Q2:The continuity of Gaussian properties (colors, rotations, etc)** We appreciate the reviewer's insight on the continuity of Gaussian properties (e.g. colors, rotations). We will seperately discuss all the Gaussian properties here. 1) **Probability.** As acknowledged by the reviewer, the Gaussian probabilities are indeed continuous in space. 2) **Color.** As detailed in Sec.3.1, the Gaussian colors are modeled using a series of spherical harmonics coefficients. Actually, these coefficients tend to be similar for nearby Gaussians in space, as they often share similar colors. For instance, the wings of an airplane typically have consistent coloring, resulting in only slight variations in the spherical harmonics coefficients for nearby Gaussians, which in turn creating a continuous color field. At the junction of two different colors (e.g., between the wing and fuselage of an airplane), the color attributes transition smoothly from one to the other by learning a continuous change in the spherical harmonics coefficients, enabled by the powerful variational auto-encoder network. 3) **Scale.** We acknowledge that scale is the most challenging property to learn in a continuous field. To address this issue, we implemented a regularization strategy, as detailed in Sec.B.1 (Gaussian Splatting Data Preparation) of the Appendix. Specifically, we observe that optimizing 3DGS freely may often lead to some extremly large Gaussians. This will lead to unstable training of the Gaussian VAE and latent diffusion models, further affecting the generative modeling results. Therefore, we clip the scales at a maximum size of 0.01 to avoid the abnormal Gaussians. Though the simple regularization on Gaussian scales, DiffGS is then capable of learning continuous scales. 4) **Rotation.** In practice, we found that the continuous learning of Gaussian rotation is strongly related to the learning of Gaussian scale. We observed that when applying the effective regularization strategy for Gaussian scales, Gaussian rotations can also be learned continuously during Gaussian optimization and VAE training. This observation is further supported by the success of TriplaneMeetGaussian in continuously modeling Gaussian rotations, where a similar regularization is applied. While this detail is not explicitly mentioned in the TriplaneMeetGaussian paper, we found evidence of this approach in their code. If the reviewer is interested, please refer to L169-172 of `TriplaneGaussian/tgs/models/renderer.py` in the code of TriplaneMeetGaussian. The key parameter `cfg.clip_scaling` which controls the scale clipping threshold can be found in L148 of `TriplaneGaussian/config.yaml` in the code. 5) **Opacity.** The nearby Gaussians contains similar opacities during the Gaussian optimizations. Therefore, the opacities of nearby Gaussians change continuous, similar to the colors. We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Best regards, Authors
Thanks for the discussion! Did our additional responses address your concerns?
Dear Reviewers, As the reviewer-author discussion period is coming to a close, we kindly ask if our rebuttal and the additional explanations have adequately addressed your concerns. If it is not the case, we are looking forward to taking the last minute to make further explanation and clarification. Best regards, Authors
Decision
Accept (poster)