Expressive Gaussian Human Avatars from Monocular RGB Video

Summary

The paper focuses on improving the expressiveness of digital human avatars, particularly through detailed hand and facial expressions, learned from monocular RGB video. The main contributions are: - SMPL-X Alignment improves the alignment of the SMPL-X model with RGB frames and aids in accurately recovering avatars. - Context-Aware Adaptive Density Control Strategy adjusts gradient thresholds to handle the varied granularity across different body parts, enhancing the expressiveness of the avatars. - Feedback Mechanism that predicts per-pixel confidence to better guide the learning and optimization of 3D Gaussians. Overall, the paper presents a comprehensive framework that enhances the realism and expressiveness of digital human representations, validated by substantial quantitative and qualitative experiments.

Strengths

The paper is written in a clear and understandable manner. Comprehensive ablation studies and clear visualizations help to demonstrate the effectiveness of different components of the method. The paper also compares the new method with previous methods and shows that it achieves better performance. It is commendable that the authors added SMPL-X to existing methods for fair and thorough comparisons.

Weaknesses

- One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works [1]. - The paper lacks a comparison to recent methods such as Splatting Avatar [2]. Including this comparison in Table 1 can help to distinguish the improvement made by the proposed approach. - There is an inconsistent ablation study across the datasets. Table 2 should illustrate how the alignment affects the baseline method by comparing scenarios with and without CADC, CL, and alignment. Similarly, Table 3 should also show the method with CADC and CL included. - PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods. - From Table 2, it seems that CL does not have a significant impact. This should be addressed or clarified to understand its role in the framework. It would be good to provide more qualitative visualization to demonstrate its effectiveness. [1] (Liu et al., 2024) GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos. [2] (Shao et al., 2024) SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.

Questions

- **Project Page for Visualizations:** Is there a project page available that shows visualizations of the rendered method, such as a 360-degree rotating video of the generated person and animation to better illustrate the quality? Specifically, could visualizations zoom into detailed areas such as the hands and face? - Is e and λt in Equation 14 learned or fixed? If learned, it would be advantageous to list the values to demonstrate that the learned parameters effectively handle different resolution details for different body parts. - Discussing failure cases and outlining potential future work would provide a more comprehensive view of the method's limitations and areas for improvement. - Information on the optimization time compared to previous methods and the resource requirements for running the proposed method would be valuable for assessing its efficiency and scalability.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

While the use of CADC is interesting, the other two contributions are of limited novelty. SMPL-X usage and alignment have been incorporated by previous works. The improvement of CL is not significant. The addition of project page for visualizations would be helpful to demonstrate the improvements made by the methods. In addition, more quantitative results should be added to Tables 1, 2 and 3.

Summary

The paper introduces a 3DGS-based human avatar generation method from a monocular RGB video. Specifically, the authors first optimize the SMPL-X model to better align with the RGB frames. Then, they propose an adaptive adjustment method for different body parts and use per-pixel confidence to guide 3DGS. It is reported that the proposed method achieves state-of-the-art performance. Visual results show fine-grained hand and facial details.

Strengths

1. Generating the human upper body videos is a critical task. It is more expressive and more challenging than facial generation. This method overall demonstrates fine-grained generation details. 2. The existing SMPL-X model has alignment issues when estimating the human body. The optimized SMPL-X model in this method significantly improves the alignment between SMPL-X and the RGB frames. 3. The paper is overall easy to follow, and the experimental results demonstrate the method's superiority in numerical metrics.

Weaknesses

1. In the paper, the confidence scores for CL are all learned from the rendered results. How can the effectiveness of this confidence be ensured? In Table 2, the face area without CL also shows better results. 2. [11] also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it. 3. The paper has many writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., $\Pi_K$ in Eq. 11), and some missing punctuation in formulas.

Questions

1. What are the specific requirements for videos in terms of the method, such as the length of the video and the range of camera angles? 2. How does the method perform when the magnitude of the action being driven is out-of-distribution? 3. What is the inference speed of the method?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

This paper has involved discussions about limitations and broader impact.

Summary

The paper presents EVA, a model that can recover expressive human avatars from monocular RGB videos. The primary focus is enhancing fine-grained hand and facial expressions using 3D Gaussian Splatting and the SMPL-X model. EVA introduces three key contributions: a plug-and-play module for better SMPL-X alignment, a context-aware adaptive density control strategy, and a feedback mechanism for optimizing 3D Gaussian learning. Extensive experiments on two benchmarks demonstrate EVA's superiority in capturing detailed expressiveness compared to previous methods.

Strengths

The paper introduces three significant innovations: a plug-and-play alignment module, a context-aware adaptive density control strategy, and a feedback mechanism for 3D Gaussian optimization. 2. Extensive quantitative and qualitative evaluations on two benchmarks show EVA's effectiveness in capturing fine-grained details, particularly in hand and facial expressions. 3. The proposed method addresses the challenging task of creating expressive avatars from monocular RGB video, which has practical applications in VR/AR, movie production, and video games. 4. The paper comprehensively explains the technical approach, including SMPL-X alignment, Gaussian optimization, and adaptive density control.

Weaknesses

1. The proposed method's complexity might pose implementation challenges for those not well-versed in the field. Simplifying some aspects or providing more intuitive explanations could help. 2. While the method shows superior performance on the provided datasets, the generalizability to other types of monocular videos or environments is not thoroughly discussed. 3. Although the paper compares EVA with previous SOTA methods, it would be beneficial to include more diverse baselines or variants to demonstrate the robustness of the proposed approach.

Questions

1. How does EVA perform with videos that have significant occlusions or low lighting conditions? 2. Are there any specific preprocessing steps required for the input monocular RGB videos? 3. Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

1. The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks. 2. The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources. 3. While the method focuses on hand and facial details, other expressive elements like body language, clothing, and hair dynamics are not addressed.

Summary

This paper proposes a solution to generate expressive human avatars from monocular RGB videos. The main focus is to improve the expressiveness. To this end, a few ideas such as combining 3d Gaussian splatting and SMPL-X (SMPL+parametric hands), minimizing 2d reprojection error, finegrained density control, etc. There are a few similar papers in citation. To better understand the novelty of this paper, here's a high level comparison: + SMPLer-X [1]: Representation is SMPL-X only. Input is monocular RGB videos. Losses: SMPL-X keypoint losses. + GART[23]: Representation is 3d Gaussians+SMPL. Input is monocular RGB videos. Losses: photometric, perceptual. + 3DGS-Avatar [33]: Representation is 3d Gaussians+SMPL; Input is monocular RGB videos; Losses: photometric, perceptual, mask, etc; + This paper: Representation is 3d Gaussians+LBS+SMPL-X. Input is monocular RGB videos. Losses: SMPL-X keypoint losses, perceptual.

Strengths

From the high level comparison, the paper does find a unique and timely combination of 3d Gaussians and SMPL-X, which is well-motivated by improving the expressiveness. Some of the ideas such as adaptive density control are therefore new as it leverages the fine-grained details of SMPL-X.

Weaknesses

The concept of model and image alignment by minimizing reprojection losses is not new. For example, the keypoints losses (Sec. 3.2) are very common in literature [1, 30]. Similarly mask loss and perceptual loss can be found in [33]. The adaptive density control part seems novel, as it's initialized from body parts and leveraging the fine-grained topology of SMPL-X. The confidence-aware loss seems new but only provides marginal improvement as in Table 2. Lack of hyperparameter ablation. Not even in supplementary. Minor issue: `video` is countable. Please explicitly use `videos` or `a video` in the paper. Actually it's not clear how many videos of one subject are used to train one avatar.

Questions

What's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor? Please also address the novelty questions in the weakness part.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations are discussed. It'd be good to discuss the failure cases.

Summary

- Given an RGB human video, the paper focuses on photorealistic reconstruction of the human in 3D using SMPL-X and Gaussian rendering. - The main idea is to align the expressive SMPL-X with the video evidence followed by Gaussian Avatar modeling with better gaussian density control and loss. - Technical contributions: Context-aware adaptive density control, confidence-aware loss, SMPL-X alignment. - Baselines: 3DGS, GART, GauHuman. - Evaluation are done on two datasets, XHuman and UPB. Metrics: PSNR, SSIM, LPIPS. - The results show that the proposed method EVA consistently outperforms the baselines.

Strengths

- The paper is well written, organized and easy to follow. - The focus problem is of great importance for the field of human digitization. - The ablative studies are informative, giving us insights into the importance of the technical components.

Weaknesses

- Weak technical contributions: In comparison to GauHuman, CVPR 2024, the proposed method EVA makes three changes. 1) Context-aware adaptive density (CADC), 2) Confidence-aware loss (CL), 3) SMPL-X alignment. CADC is a nice idea for human modeling, and is inspired by the original 3DGS paper. However, it is a slight change to the originally proposed heuristic. CL is frequently used in the field of pose-estimation and 3D modeling (eg. DUST3R, CVPR 2024). SMPL-X alignment is very simply mesh fitting to 2D keypoints and is well explored in SMPLify-X, CVPR 2019 and its related works. Overall, the main technical ideas proposed lack technical novelty. - XHuman over ZJU_MoCap for evaluations: Is there a reason why metrics were reported on XHuman and not ZJU-MoCap dataset? The missing metrics on the popular ZJU-MoCap benchmark makes it hard to fairly compare EVA with existing. I am guessing this because ZJU-MoCap does not have SMPL-X annotations. In this context, can we use the same setup as UPB and still evaluate using predicted SMPL-X parameters?

Questions

Given the lack of technical novelty, it is important to establish the empirical usefulness of the proposed method. In this context, I am asking for a fair evaluation with prior methods (mostly using SMPL) on popular benchmarks.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Thanks for the responses

Thanks for the detailed responses. Given the overall quality of this submission, I will recommend the acceptance of this paper.

Response to Authors

Dear Reviewer, The discussion period with the authors is coming to a close in less than 24 hours, at 11:59 PM AOE Aug 13, 2024. There is still time to review the authors' response to your questions and engage in a discussion with them. It would great if you could acknowledge having read the authors' response and update your rating if it changes your mind. Best, AC