XAGen: 3D Expressive Human Avatars Generation

Summary

The paper proposes a generative model for 3D expressive human avatars. Based on the backbone of the recent works, tri-plane 3D feature representation, volumetric rendering, transformation from pose space to canonical space, parametric body models and GAN, the authors propose to improve the expressive of the model by representing the hand and face with extra tri-planes and supervising them with extra adversarial loss. The result achieves state of the art performance according to various evaluation metrics on various dataset. Moreover, one can animate the hand and jaw of the generated avatar, surpassing most SOTA approaches. Two practical applications of such a generative model are demonstrated in the paper.

Strengths

1. The paper presents a state of the art generative model for 3D clothed human avatar with expressive control. 2. The paper is solid, well written and easy to follow. 3. The evaluation is extensive and the results are convincing. 4. Two practical applications are demonstrated.

Weaknesses

1. Lack of technical novelty, multi-tri-plane and part focused supervision are quite weak when claimed as novelty. But I'm relatively ok with this since the performance and the control of the model surpass the SOTA. Although in my opinion, this paper fits much more for SIGGRAPH/Asia, where technical novelty is less expected compared to NeurIPS, I believe the overall quality of the submission clearly reaches the acceptance standard for NeurIPS. 2. While I appreciate the effort on making consistent and good format of references (I notice the authors use consistent form of names of the same conference across the reference, which is rare nowadays in most of the submissions), there is still room to improve the reference, such as a. Capitalization of words. For example: 3d -> 3D, Icon: implicit... -> ICON: Implicit..., Avartargen: a 3d... -> AvartarGen: A 3D..., b. the URL in the references is not necessary. c. [65] In ECCVw, 2023 -> In ECCV Workshops, 2022 3. In the supplementary video, a. the voice over contains environment noise and echoes. It would be great to remove them. b. the bottom left animation has a white background, occluding the slide contents. It could be better if those animations were rendered with transparent background. c. the demonstration of controlling jaw motion and hand poses are a bit long and not eye catching. It would be more appealing if they can be shortened a bit and shown with a close-up / zoom-in view. 4. On the project page, the RGB video and the geometry visualization are asynchronized regardless of refreshing the page. 5. Suggestion. Please consider incorporating some concurrent work into related work for the updated version to make the paper more inclusive, for example: Chupa (https://snuvclab.github.io/chupa/), a generative model for 3D clothed human geometry; and SCARF (https://yfeng95.github.io/scarf/), creating an expressive avatar from monocular video.

Questions

The submission is in a good state, method clearly stated, experiment is extensive and convincing. I don't have further questions to the authors regarding reviewing this paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations and societal impact are discussed.

Summary

The paper proposes a method for the generation of high quality, articulable 3D avatars of humans. The method proposed builds on top of the 3D GAN framework which has been used to learn to generate 3D articulable human bodies from collections of 2D images of humans. Additionally, the method adopts the proposed high-level methodology of generating bodies in a canonical pose and then using an explicit deformation guided by a parametric body model in order to render images in a desired pose. The novel contribution of the method is unifying this generation architecture not only for bodies, but for faces, bodies, and hands, and demonstrating that generating, articulating, and discriminating these body parts individually results in a higher quality result than handling them all simultaneously. The paper describes the architecture which accomplishes this, and demonstrates that this results in a state-of-the-art generation quality for 3D human avatars. The contributions of modeling and applying losses to these body parts separately are ablated, demonstrating that they are responsible for improvement in the quality of generated avatars. AFTER REBUTTAL: I have read the authors' rebuttal. I believe the additional comparisons provided for the hand and jaw quality address my weakness there. As I mentioned, I still have some questions about the novelty of each of the components, but I think the method performs state-of-the-art overall and combining all of these components into a working method is important for the community. Thus, I am leaning positively.

Strengths

In my opinion, the main strengths of the paper are that: 1. The paper is well written and is clear to read and follow, and presentation is extremely polished. As a result, I feel like the method will be impactful and those working in the field of generative 3D articulable avatars will want to build on top of it. 2. The proposed method demonstrates state-of-the-art results. The evaluations are extremely convincing both qualitatively and quantitatively, and compared to the baselines EVA3D and AvatarGen which are currently generating the state-of-the-art, I believe the generated results are significantly better. This is extremely important in pushing towards photorealistic generative 3D human avatars. 3. The method is ablated well. The main contributions: separately modeling faces and hands, and using a separate discriminator for each of them, are both ablated in a very clear way showing how they improve the performance of the method. This is extremely important to understand that the contributions proposed are actually responsible for the increase in quality, rather than just hyperparameter tuning.

Weaknesses

In my opinion, there are two weaknesses of the paper. 1. The evaluation accuracy of the facial/hand articulation is not entirely convincing. - While Table 2 shows that there are improvements in “Jaw” and “Hand” when a facial and hand pose estimator is applied to the generated results, it is only compared to the modified AvatarGen with SMPL-X (which it outperforms). In order to actually ground these numbers with a baseline, it would be insightful to apply these pose estimators to the results of EVA3D and ENARF. I understand these methods don’t model hand or facial poses, but it would be an important comparison to understand how much improvement is gotten from even modeling these at all. - Quantitative comparisons don’t ever show multiple different identities with modified facial or hand poses (for example, in the supplementary video). Because of this, I don’t know if the examples provided (for example, mouth open), are cherry-picked results or consistent. It would be immensely helpful to show a result where the facial/hand pose is held constant in a deformed state (for example, mouth open), and then the identity latent space is walked through. 2. The amount of method novelty is relatively limited. While the results are very nice and this is an important contribution, the method is based off of the same framework as ENARF/GNARF/EVA3D/AvatarGen where bodies are generated in the canonical pose and then deformed, and the tri-plane representation is used for volume rendering. While explicitly modeling hands and faces is important, it seems like a combination of methods like GNARF which have already been applied to heads, and just putting them in the same model. The additional results with text-driven generation or audio-driven manipulation are very cool, but are not technically novel and are instead a combination of open-source models (and can be applied to any other generative articulable 3D representation).

Questions

1. It is mentioned that the left and right hand are represented with the same set of tri-planes but mirrored. Does this limit the capacity in the generation, i.e. when there are different accessories on either hand? 2. Is truncation used when sampling the generated results from the method? Is it used for sampling for the baseline methods, such as AvatarGen and EVA3D? This should be standard for evaluating qualitative results. 3. I assume this is using the same representation as AvatarGen, but why is it not described in the methodology. Is the representation a NeRF? An SDF? How is volume rendering done?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

The limitations of the method have not been explicitly addressed in the paper, although they are addressed in the supplementary information. In my opinion, including some of the limitations, especially surrounding the quality of the datasets and SMPL-X estimators (and how this actually affects the quality of the generated results) would be insightful (potentially more-so than the added applications, such as audio-driven acting or text-driven generation).

Summary

The authors proposed novel part-aware sampling and feature parametrization strategies to improve the fidelity of the avatar model, especially for smaller body parts. These techniques s enable the efficient learning of diverse fashion shapes with a focus on the hand and facial details. Through experimental evaluation of several benchmarks, the method surpasses all baselines including the recent work AG3D for a huge margin. The results emphasize the necessity of introducing discriminator techniques. Moreover, the authors demonstrate control with text and audio-generated poses as a possible application of their model.

Strengths

I appreciate the authors' effort in presentation results and getting fair comparisons with baselines. - Undoubtedly, the proposed technique for discriminative learning is important and interesting - easily transferable to other models, which is clearly well studied - The authors suggest using separate representations for significant parts - hands, face, and body as a simple technique to drastically improve the capacity of the model - Comparison presented on three data unrelated to the training corpus.

Weaknesses

- some important architectural choices are not studied. The original idea of sepration face and body doesn’t have cleare explanation or intuition to separate face from the body for features (not in discriminator) - The ideas lying in the core of the method are quite unpretentious, which is not bad but it decreases the novelty and impactful of the model itself. - ethical and limitation sections are missing in the main text - Animations are only shown for synthetic pose sequences. Try to extend it with real sequences (e.g. AMASS), the body movements can improve the credibility of the method. - No images from the competitor, who utilizes the part-based discriminator as well - AG3D - ask authors to sample more images for you. - SMPL-based deformation model will affect loose clothing that is not covered in the main text

Questions

- Why do you condition the discriminator for shape and expression (and not pose to handle hands/skeleton deformations or just clear this moment in the main text)? Is it important if so, could you add reference numbers for that? - Is it possible to compare the methods for human preferences score, since metrics here are not ideal measures? At least a more careful qualitative evaluation can be a good extension it is difficult to see the difference in Figure 4. Is it possible that FID results here do not correlate well with visual ones? - What if the size of the face and hands tri-plane will be downsampled twice more? - Does geometry contain any facial attributes related to expression or fingers? - What is happening with loose clothing since you have quite a simple deformation model? - How good is this part-based scheme for image inversion - do you have any results of intuition in comparison with EVA3D or AvatarGen.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

It is difficult to handle wrinkles or accurate textures as well as loose cloth. Moreover, the demonstrated method is less efficient due to the extended representation. Due to the fact that the model is trained without video data, it is not possible to definitively determine the correspondence of pixels can be incorrect. The trained model has a huge bias to fashion poses due to training data that should be taken into account with all conclusions from the model.

Summary

The paper presents a framework for human 3D avatar generation. The whole model is trained on a set of 2D images, thus can support large variations in terms of shape. The framework is based on EG3D with several important modifications: 1) Incorporation of the inverse LBS, that will deform the canonical representation of the object. 2) Modeling hands and face with separate triplane branches. 3) Additional part based discriminators for face and hands. The paper demonstrates good results, that include modeling of the face and hands motion. Although the motion of these parts are rather weak and contain a lot of artifacts. POST Rebuttal Summary: The answers in the rebuttal makes sense, in the rebuttal authors provided additional results on sensitivity to SMPL-X errors. To this end I increased the rating by one, and encourage the authors to improve the presentation quality of the method and results.

Strengths

- The paper proposed a meaningful framework that combines several existing ideas into a single unified 3D framework. - The paper demonstrates good results for the full body modeling. - The paper contains rigorous evaluation of the different modeling aspects, as well as detailed comparison with sota methods.

Weaknesses

- The quality of presentation is not the best, the method is presented in such a way that it looks like a combination of InsetGan[1] and EG3D. Also visual results presented only on a single dataset, or it is not clear where the results on other datasets. - Some architectural choices seem questionable. For example why nearest neighbor LBS, was used, instead of the more advanced LBS approximation from HumanNerf[2] or even more advanced SNARF[3]? Also EG3D, because the upsample produces visible inconsistencies in the generated results, why a more advanced training scheme, such as EpiGRAF is not used? The current LBS approximation will struggle with loose clothing that is not well represented with SMPL-X. - The quality of hands and face is extremely poor. Why it is bad is not analyzed in the paper. Is this a dataset issue? Is it possible to use specific dedicated face/hand datasets in addition to human bodies dataset, and training only face and hand discriminators with them? Or is it an SMPL-X fitting problem? Again is it possible to use more specific hand/face models to improve this aspect? [1] InsetGAN for Full-Body Image Generation https://arxiv.org/pdf/2203.07293.pdf [2] HumanNeRF:Free-viewpoint Rendering of Moving People from Monocular Video https://grail.cs.washington.edu/projects/humannerf/ [3] SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes https://arxiv.org/abs/2104.03953 [4] EpiGRAF: Rethinking training of 3D GANs https://arxiv.org/abs/2206.10535

Questions

Questions listed in weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

- Inconsistency in the results. - Poor quality of the hand and face regions. - Not very good presentation. Visual results only on a single dataset, no visual comparison with other methods in supplementary videos.

Summary

This work addresses the problem of 3D full body avatar generation, going beyond prior work primarily on more detailed hand and face generation quality and controllability. The pipeline comprises a 3D-aware GAN where the generator generates tri-plane feature maps from noise vector, followed by a 3D decoder that generates the human geometry represented by SDFs, in a canonical pose. The canonical SDF is then posed (with the help of SMPL-X body model) and rendered to be discriminated about the image discriminators. The key contributions of the work is a complete pipeline to model the face and hand regions separately, equip them with controllability (e.g. hand pose and facial expression), via the body model, and finally unify them to the full body generation. This effectively addressed the limitation of prior work, namely the compromised hand/face quality due to their relatively small region in most training images. Extensive experiments show the edge of this work over other recent human generators. The authors also demonstrated the usefulness of a full-body avatar generative model in the applications of test/audio-driven avatar synthesis.

Strengths

- This work provides a complete pipeline for controllable 3D avatar generation, where the components in this pipeline are combined in an intuitive and effective way. Speaking of novelty, although the tri-plane-based 3D GANs are common nowadays and the body-model-aided reposing mechanism is also well-known, I can't come up with other work in this line that well fixes the problem of hand/face quality and controllability. Therefore, the full method makes a good technical contribution to the avatar generative modeling. - Experiment section is strong, and the results are impressive. It incorporates major recent methods on controllable avatar generation and show a clear performance edge in terms of the general coherency of geometry, rendered image fidelity, pose controllability and sharpness on the face/hand region, validating the effectiveness major technical contribution. - The paper is well-written, being clear and easy to follow. The main paper is mostly self-contained, and the SupMat is also very thorough.

Weaknesses

- The generated geometry has limited resolution, are prone to artifacts, and in most of them, the face/hand region are missing details that are present in the textured renderings. Despite the careful handling of hand/face regions, the geometry of these areas generally show a lower quality than the texture. In other words, although the work targets geometry generation, the model often "covers up" what should be geometric structures with texture. - Although the generated avatars can indeed be animated (via SMPL-X) as claimed, there seems to be no mechanism that ensures the physical correctness (e.g. semantics of body parts and cloth) or visual plausibility (e.g. pose-aware cloth deformation) of the animated results. For example, when the generated character has self-contact (e.g. Fig 3, lower right "Ours", the subject that touches the face), the contact area are merged as if they are connected (please correct me if wrong); consequently, when animating this subject with novel poses (e.g. arms-open), visual artifacts can appear.

Questions

- The hand, face and body feature maps are generated together in a channel-stack manner. Intuitively, the features of these three different parts are not spatially aligned, and generating them with a same generator might not have other benefits than saving computational cost. Is it investigated whether training separated generators for these 3 parts brings better quality? - For the face region specifically, AvatarGen also employs a dedicated descriminator, similar to the one used in this work, and yet performs poorly in terms of plausibility of facial detail and expressions. What is the key differentiator in the method that brings such salient improvement?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The techinical limitations and potential societal impacts are discussed in detail in the SupMat, I really appreciate that.