GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats

Summary

The paper presents a 3D-aware GAN (3DGAN) framework using a 3D Gaussian Splatting model. Given a collection of single view 2D images (e.g., FFHQ, AFHQ), the proposed method can train 3D aware images of comparable quality to SOTA NeRF-based method, but with a much faster rendering speed (up to 100x) during inference, thanks to the efficient 3DGS representation. The paper achieves strict 3D consistency as all the pixels are rendered with 3DGS without relying on post 2D super resolution (EG3D). This is among the first works (e.g., Gaussian Shell Maps in CVPR2024) that introduce the 3DGS in 3DGAN framework. In details, the paper uses hierarchical gaussian representations, and the 3D representations are learned in a coarse-to-fine manner. To regularize the generation of 3D Gaussians the paper introduces anchor gaussians which will create the next level gaussians, but the anchor gaussians itself were only used for regularization. Additional regularization losses are applied to the coarsest level gaussian anchors to regularize their positions. In terms of the generator architecture, the paper borrows similar structures from StyleGANs, similar to other 3D GANs but introduces some attention-based MLP architectures. The paper also introduces the layerscale layer, which is initialized to be zero to stabilize the positions of the gaussians in the early stage of the training. Coincidentally, the same strategy was also used in GGHead.

Strengths

-The paper presents a 3D GAN framework using a 3D GS representation. The method achieves strict 3D consistency and a significantly improved speed while achieving comparable FID scores to SOTA NeRF methods (e.g., Mimic3D) -learning 3D representations from a single view in the wild image collection is an important research area and this area has improved significantly due to the improved computational efficiency of 3D representation (e.g., triplane in EG3D). The use of 3DGS in 3DGAN will likely boost the overall quality of 3D learning from in the wild 2D images, and the similar method could be applied in the future to learn other domains (e.g., objects, full body etc). -The paper seems to have sufficient numerical and qualitative comparisons. -The paper is easy to follow.

Weaknesses

-The related work section is extremely short and is lacking a lot of relevant citations. Please look at recent papers in the similar area (e.g., **GGHead: https://arxiv.org/abs/2406.09377**, **WYSIWYG: https://research.nvidia.com/labs/nxp/wysiwyg/**), and properly cite all the related works. This is a major issue. -Gaussian Shell Maps is also a highly relevant work (CVPR 2024), the code is available here: https://github.com/computational-imaging/GSM -Most technical details are provided but some important details seem missing (see below)

Questions

-On the most important technical contribution, I did not fully understand why the paper needs to introduce anchor gaussians. Couldn't the same regularizations (if they exist at all) applied to the rendering gaussians themselves? -In early 3D GAN research (e.g., pi-GAN), they used fully MLP-based architectures to learn unconstrained 3D space, but later EG3D showed triplanes are a more efficient 3D representations. Did the author consider the use of triplane? -How many Gaussians are used in total with all layers and how sensitive the method is for the number of gaussians? I think this is an interesting question many readers would be interested in. -it seems that the position regularizations only applied to the coarsest level in Eq (10). What about scale regularizations? What about regularizations to other layers? Please clarify. -In Eq (4), why there is an additional delta s, which seems to be doing the same thing as s^{hat_l}? How is delta s < 0 enforced in the loss? This detail seems missing.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations and potential negative impacts are explained. Additional limitation could be the generated image quality is still not on par with post 2D super resolution 3DGAN baseline (EG3D).

Summary

This paper proposes a new 3DGAN based on the 3DGS representation. To maintain stable training and good generation quality, this paper introduces a hierarchical Gaussian to generate the results in a coarse-to-fine manner. Meanwhile, a reasonable transformer-based architecture is proposed to implement the hierarchical Gaussian. Moreover, the proposed Anchor Gaussian is also reasonable. The generation results in both the cat and human domains are good.

Strengths

1. The whole pipeline is reasonable and interesting. To stable 3DGAN training when using 3DGS, it is smart to utilize a hierarchical architecture to predict the parameters of 3DGS in a coarse-to-fine manner. 2. The Anchor Gaussian can disentangle the rendering and regulation well, which helps the method generate high-quality results. 3. The transformer-based architecture is suitable to implement the Hierarchical Gaussians. 4. The results are good and the FID is also comparable. Meanwhile, since the inherent advantage of 3DGS, the proposed method can generate high-resolution images with low time-consuming.

Weaknesses

1. The L_pose is not used in the EG3D or other baseline methods, and I want to know how this loss affects the training of the proposed method. Meanwhile, the pose encoder of the camera parameter Is trained simultaneously with GAN? 2. Is there a branch that generates the background regions similar to the Panohead? I find some artifacts in the generation results (i.e., some floaters). Overall, this is a good paper that introduces a reasonable 3DGS-based GAN. I tend to accept the current version. I run the code, the results are good!

Questions

see the weakness.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

see the weakness.

Summary

The paper proposes a hierarchical Gaussian Splatting for 3D GAN. The authors claim that such structure lead to stable training and fast rendering.

Strengths

* The hierarchical GS structure is interesting.

Weaknesses

* The motivation to propose such a hierarchical GS structure is not clear. As stated in line 119-122, the design is used to mitigate the instability of position/scale in GS training. It is not an intuitive solution and it is not well interpreted in the paper. The examples in Fig. 6 are too simple and confusing, please refine the figures to make it easy to understand why the hierarchical design is reasonable. * The hierarchical structure of Gaussian Splatting is not a new topic. The literature review about GS and its variants is not sufficient to support the claimed novelty. * The author claims the rendering speed as their contribution. In my opinion, the fast rendering is the characteristic of GS itself. If the authors claim the proposed hierarchical GS makes it faster, ablation and discussion are needed as evidence. As shown in Tab. 1, the selected methods for rendering time comparison are all papers published before GS.

Questions

Please refer to "weaknesses".

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No negative social impact.

Summary

The paper proposes a method to train 3D GANs using 3D Gaussians. A limitation to training 3D Gaussians in an adversarial training setup is the instability of the scale optimization, which leads to the scale explosion. To address this, the paper proposes a method of hierarchical generation of Gaussians using the Stylegan architecture. As a result, this approach can generate consistent 3D objects at higher rendering speeds. The results are comparable to the volume rendering approaches with the approach surpassing the generation speeds. The authors evaluate their method on two datasets – FFHQ and AFHQ CATs.

Strengths

1) The paper addresses the problem of scale explosion when 3D Gaussians are trained under adversarial losses. The paper addresses this problem by a hierarchical approach to the generation of these Gaussians. This would help in scaling future 3D models to higher texture resolutions. 2) The paper comprehensively compares the existing methods based on volume rendering and shows that the quality and consistency of results are comparable. The rendering speed exceeds the volume rendering methods. 3) The paper shows inversion results that have applications in various single image to 3D generation tasks.

Weaknesses

1) Why not the original resolution of 1024 for the faces? The authors claim that the generator is 100x faster. The paper does not demonstrate results with this resolution since it fixes the scale. GSMs[1] also mention the scale problem that makes it difficult to train high-frequency details. If the current method addresses the problem, does it scale well to higher resolution? 2) There are many works now employing underlying FLAME/SMPL models like GSM[1], GNARF[2] etc to model faces/bodies. This helps in natural articulation and control. This framework only supports generation with limited articulation and editing. A discussion of these frameworks with limitations and advantages would help. 3) Although the authors show the scaled-down Gaussians for visualization, there is no evidence of extraction of geometry from the models. Some works in the Gaussian domain[3] try to extract the geometry. How does it compare with Eg3d? How does running the COLMAP on the outputs look like? Are there more artefacts than the volume rendering methods? 4) There still seem to be some floating artefacts as seen in the supplementary videos. [1] Abdal, Rameen, et al. "Gaussian shell maps for efficient 3d human generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Bergman, Alexander, et al. "Generative neural articulated radiance fields." Advances in Neural Information Processing Systems 35 (2022): 19900-19916. [3] https://github.com/MrNeRF/awesome-3D-gaussian-splatting

Questions

1) What specific challenges were encountered when attempting to train or generate images at the original resolution of 1024? 2) Are there plans to demonstrate results at 1024 resolution in future work, and what optimizations might be needed to achieve this? 3) Can the architecture be adapted or extended to handle high-frequency details better at higher resolutions? 4) How does the current framework compare to models like GSM[1] and GNARF[2] in terms of articulation and control? 5) Can the integration of FLAME/SMPL models enhance the current framework's capabilities, and if so, how might this be implemented? 6) Have you attempted to extract geometry from your models, and if so, what were the results? 7) How does your approach compare with Eg3d regarding geometry extraction and handling artefacts? 8) What does running COLMAP on your outputs reveal about the quality and presence of artefacts compared to volume rendering methods? 9) What are the primary causes of floating artefacts observed in your results? 10) Are there specific scenarios or conditions where floating artefacts are more prominent?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The Authors have addressed the limitations.