Generalizable and Animatable Gaussian Head Avatar

Summary

This paper presents a method for animatable facial avatar reconstruction from a single RGB image, and the reconstructed avatar is based on 3D Gaussian splatting (3DGS) to support real-time rendering. To this end, the proposed method generates pixel-aligned Gaussian point clouds to reconstruct the identity, and use 3DMM to construct an additional set of Gaussian points to control the facial expression. The network is trained on a large scale corpus of human talking videos. After training, the network can produce reenactment results at real-time framerate for any unseen identities. Experiments show that the proposed method is able to produce plausible reenactment results.

Strengths

* This paper proposes the first generalizable 3D Gaussian-based avatar modeling method (if not taking ECCV 2024 papers into account). * The authors compare the proposed method with most recent state-of-the-art baselines (such as Protrait-v2). The comparison is comprehensive and convincing. * Results in Table 1 and Table 2 show that the proposed method consistently outperforms existing approaches under differently metrics and different settings.

Weaknesses

* My major concern is about the method design. Despite the plausible results, the proposed method is confusing and doesn't make sense to me. If I understand correctly, the reconstruction branch produces Gaussian points that are aligned with the input image, and these points keep static and unchanged when performing reenactment. Instead, the expressions are modeled through the additional Gaussian points that are attached to the 3DMM surface (Line 187-188), which means these expression Gaussian points should play a major role in expression control. However, according to Figure 7, the Gaussian points produced by the reconstruction branch actually play a major role in facial rendering, while the "expression Gaussians" seem to be much less important. That doesn't make sense to me: how could the static points produced by the reconstruct branch model the dynamic expressions? When overlapping the reconstruction Gaussian points with the expression ones, how does the proposed method resolve the potential conflicts of these two point sets, such as different lip positions? * The so-called dual-lifting technique looks straight-forward and trivial to me. According to Sec.3.1, this module produces two pixel-aligned Gaussian point sets, one for the visible surface and the other for the invisible surface. However, using two set of points (visible+invisible, or front+back) to model a complete shape is not a new idea; it has been used in 3D human reconstruction like [a] many years ago. Although such an idea has not been proposed in the form of 3DGS, I don't think it is novel enough to be claimed as a technical contribution. * It is unclear whether the baseline methods are trained on the same dataset as the proposed method or not. In Line 244 of the main paper, the authors mention that they "use the official implementation", but it is still unclear whether the authors use the pretrained network weights or retrain these baselines using the same dataset. [a] Gabeur et al. Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images. ICCV 2019.

Questions

One minor suggestion: it would be better if the authors could provide some qualitative comparison against state-of-the-art methods in the form of dynamic reenactment videos and highlight the difference/advantages of the results.

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have discussed the limitation and potential social impact in Sec.5 of the main paper and Sec.E & F of the supplemental document.

Summary

The paper proposes a method to achieve one-shot head avatar animation with 3D Gaussian Splatting (3DGS). With 3DGS, the papers show high-fidelity animation with fast inference speed. To solve one-shot 3DGS reconstruction, the paper proposes a dual-lifting method with 3DMM regularization. Experiments justify the model designs and show the proposed approach achieves SOTA performance.

Strengths

1. The paper proposes a technically sound approach for solving the one-shot animatable head avatar problem using 3D Gaussian Splatting (3DGS), fully utilizing the pre-trained 2D network to obtain cues for lifting. 2. The quantitative results are promising, and the qualitative results are sufficient to demonstrate the robustness of the approach.

Weaknesses

1. The visual results of the method seem to be overly smoothed to some extent, making the results less realistic compared to some baselines (e.g., Portrait4Dv2). The authors may need to provide additional explanation for this and discuss why the method achieves better qualitative results than other methods. 2. The approach is highly dependent on 3DMM but lacks a discussion about its impact. What if the estimation fails? How does the estimation accuracy affect the final performance (both training and inference)? If using other designs without 3DMM, what would be the method's performance? 3. There should be some failure cases to show the limitations of the work.

Questions

1. The current ablation study may not be sufficient to validate the dual-plane lifting. Is it possible to conduct a study by removing the reconstruction branch to better demonstrate the importance of dual-plane lifting?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors discuss their "Limitations" and provide an "In-Depth Ethical Discussion" proposing several measures to prevent these ethical risks.

Summary

The paper introduces a novel framework, GAGA, for one-shot animatable head avatar reconstruction from a single image. The key innovation is the dual-lifting method, which generates high-fidelity 3D Gaussians that capture identity and facial details by predicting lifting distances from the image plane to 3D space. This method leverages global image features and a 3D morphable model (3DMM) to ensure accurate expression control and real-time reenactment speeds. The model can generalize to unseen identities without specific optimizations. Experimental results demonstrate that GAGA outperforms previous methods in terms of reconstruction quality and expression accuracy. The main contributions include the introduction of the dual-lifting method, the use of 3DMM priors to constrain the lifting process, and the combination of 3DMM priors with 3D Gaussians for efficient expression transfer, allowing high-quality real-time rendering.

Strengths

- The proposed dual-lifting of approach is interesting and enables the hallucination of unseen areas and resolves ambiguity of front/back 3D Gaussians. - The combination of expression Gaussians and dual-lifted Gaussians allows flexible expression control. - The method outperforms various baselines in a wide range of metrics. The rendering speed also significantly outperforms prior methods. - The paper is clearly written and easy to follow.

Weaknesses

- The Gaussians from the expression branch only use the vertices of the 3DMM model, which can be inaccurate and unable to model fine facial features. - It would be nice to show more side-by-side video comparison with baselines. Many baselines look similar qualitatively and it’s hard to tell the improvement without highlighting the difference.

Questions

- Why not also predict position offsets for the expression Gaussians? To keep expression structures, regularization can be used. - Would the two sets of Gaussians (expression & dual-lifted) contract each other during expression? - Why not bind the lifted Gaussians to the 3DMM vertices and animate together?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper acknowledges several limitations. First, the model may generate less detailed reconstructions for unseen areas of the face, such as the back of the head or the interior of the mouth, which are not visible in the input image. Second, the 3DMM-based expression branch cannot control parts of the head that are not modeled by 3DMM, such as hair and the tongue.

Summary

This paper presents "Generalizable and Animatable Gaussian Head Avatar" (GAGA), a method for one-shot animatable head avatar reconstruction using 3D Gaussians. Unlike existing methods that depend on neural radiance fields and require extensive rendering time, GAGA employs a dual-lifting method to generate high-fidelity 3D Gaussians from a single image in a single forward pass. The approach integrates global image features and 3D morphable models to control expressions and achieve real-time reenactment speeds. Experiments demonstrate that GAGA outperforms previous methods in reconstruction quality and expression accuracy.

Strengths

1. The dual-lifting method for reconstructing 3D Gaussians from a single image is novel and effective, allowing for high-fidelity reconstruction. 2. Efficient resource utilization during training and inference makes it practical for real-time applications, which is a significant improvement over existing methods that are often slow. 3. The model can generalize to unseen identities without specific optimizations, broadening its applicability. 4. Experimental results show superior reconstruction quality and expression accuracy performance compared to state-of-the-art methods.

Weaknesses

1. The reliance on 3D morphable models for expression control may limit the model's flexibility. 2. The 3DMM identity and expression parameters are not entirely decoupled, potentially impacting cross-identity reenactment identity consistency. 3. The method may generate less detail for unseen areas and has limitations in controlling regions not modeled by 3DMM, such as hair and tongue.

Questions

1. How sensitive is the model to variations in the input image quality and lighting conditions? 2. How robust is the model when applied to images with significant occlusions or extreme facial expressions? 3. What specific measures were taken to ensure the diversity and quality of the training data?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The model may produce less detail for unseen areas, which can affect the realism of the generated avatars in dynamic scenes. 2. The expression control is limited by the constraints of the 3DMM, which does not model elements like hair and tongue, leading to potential inaccuracies. 3. There is a slight compromise in identity consistency of cross-identity reenactment due to the partial decoupling of identity and expression parameters in the 3DMM. 4. While the method achieves real-time speeds, the dual-lifting and neural rendering processes add complexity to the implementation and may require significant computational resources for training.

Thank you

Thank you for the detailed response and additional experiments. I will maintain my initial score of weak accept.

Author Rebuttal

Dear Reviewers, The authors have provided detailed responses to the questions that you raised. I strongly encourage you to please read the authors' responses and acknowledge that you have read them before the AOE 11:59 PM (Aug 13, 2024). Best, AC

Thanks for the rebuttal.

Thank you for the rebuttal. After reading the comments from other reviewers and the reponses from authors, I would keep my rating as borderline accept.

Response to Authors

Dear Reviewer d3t7, The authors have provided a response to the questions you raised in your rebuttal. The discussion period with them is soon coming to a close, in less than 24 hours at 11:59 PM AOE Aug 13, 2024. It would be great if you could acknowledge that you've read the authors response, engage in any further discussion with them and update your score if needed. Best, AC