Summary
This paper introduces a novel amortized Text-to-3D Gaussian generation method, enabling the creation of 3D Gaussian structures from text prompts in a single forward pass. The proposed approach incorporates two key innovations:
- Anchor-based Deformation:
Traditional 3D Gaussian representations require numerous Gaussian points to model complex scenes, making direct generation challenging. To address this, the authors introduce a strategy where a small number of anchor points are fixed and subsequently deformed through a text-conditioned transformer. These deformed points act as the central points for generating the 3D Gaussian structures.
- Text-guided Triplane Generator (TTG):
The paper further proposes the use of a Text-guided Triplane Generator (TTG) to create a triplane structure. The anchor points are then utilized to query Gaussian features from the generated triplane.
To train the models, the Score Distillation Sampling (SDS) method is applied to rendered images, following the practice used in previous amortized Text-to-3D models. This ensures effective training of the networks. The authors conducted extensive experiments, demonstrating the effectiveness of their approach compared to traditional train-from-scratch Text-to-3D Gaussian methods.
Strengths
This paper consist of following strengths:
- **Introduction of 3D Gaussians to Amortized Text-to-3D Generation:**
This paper is the first to apply 3D Gaussians to the amortized Text-to-3D generation task, demonstrating higher efficiency compared to traditional methods that require training from scratch.
- **Reframing the 3D Gaussian Generation Problem:**
The paper transforms the challenge of generating 3D Gaussians into a deformation problem. This approach addresses the difficulty of generating complex objects that would typically require millions of Gaussian points, making direct generation infeasible.
- **Proposal of a Triplane Generator for Spatial Features:**
A novel triplane generator is introduced to produce spatial features, which can be decoded into Gaussian attributes. The method is further refined with several specific design improvements to enhance performance.
- **Experimental Validation of Efficiency:**
Extensive experiments validate the efficiency of the proposed approach compared to original Text-to-3D Gaussian generation methods.
Weaknesses
Though this method achieves promising results through its proposed designs, the paper presents several weaknesses that need to be addressed:
**1.Novelty:**
While the paper is the first to reformulate 3D Gaussian generation as a deformation problem, this concept is not new in the field of explicit 3D generation. Similar deformation-based approaches have been explored in previous works, such as:
- Wang, Nanyang, et al. "Pixel2mesh: Generating 3d mesh models from single rgb images." Proceedings of the European conference on computer vision (ECCV). 2018.
- Wen, Chao, et al. "Pixel2mesh++: Multi-view 3d mesh generation via deformation." Proceedings of the IEEE/CVF international conference on computer vision. 2019.
Since 3D Gaussians represent a novel but explicit 3D format, the deformation approach is straightforward to apply, suggesting that this work is more of an incremental adaptation than a fundamentally novel idea.
It is accessable to bring an old idea to a novel field. However, another critical issue lies in the representation ability. While Gaussian splatting typically involves millions of points to accurately capture scene details, this paper proposes using a fixed, smaller number of anchor points. The lack of discussion on how many anchor points are required for convincing results raises concerns about the expressiveness of the generated models. If the number is reduced too much, the method may struggle to represent complex scenes effectively. The authors should address this limitation by providing a more thorough analysis and justification for their approach.
**2.Network Structure:**
The network architecture closely resembles the one proposed by Zou et al. in:
- Zou, Zi-Xin, et al. "Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers." Proceedings of the IEEE/CVF CVPR. 2024.
Both networks feature a triplane decoder and a point-based querier. Although the authors claim to introduce improvements, the paper does not provide an ablation study to evaluate the impact of these modifications. A more in-depth discussion and comprehensive ablation studies are necessary to demonstrate the value of these improvements.
**3.Experiments:**
The experimental results focus only on comparisons with text-to-3D Gaussian generation methods. However, since the paper's task aligns more closely with amortized Text-to-3D generation, it should also be compared against those methods to ensure a fair evaluation.
Additionally, as mentioned earlier, an ablation study is needed to assess the effectiveness of the network design choices. This would clarify the role of individual components and provide insights into which modifications contribute to the improved performance.
**4.Writing:**
The paper introduces new terms, such as "Triplane Generator Division" and "Coordinate Shortcut," in ablation study, but fails to explain them clearly. As a result, it becomes difficult to understand the exact meaning and purpose of these concepts. Providing detailed descriptions of these definitions would improve clarity and make the paper more accessible to readers.
Questions
As noted in the Weaknesses section, I believe the authors should clarify the motivation behind their approach and the network structure similarty with triplane meet gaussians. Therefore, I am giving a rating of 3. If the authors address these points, I would be open to raising my score.