DreamWaltz: Make a Scene with Complex 3D Animatable Avatars

Summary

The paper presents an approach for creating animatable avatars from text prompts. It builds on DreamFusion and makes it articulated by incorporating articulated NeRF and SMPL body model. It also replaces the vanilla text-to-image model (StableDiffusion) with ControlNet to introduce 3D consistent SDS loss. The performance is evaluated using a user study where the proposed method is shown to outperform existing methods.

Strengths

- The paper addresses the challenging problem of creating neural avatars from text prompts. - The paper demonstrates that DreamFusion can be extended to articulated humans. - The proposed method is intuitive and makes sense, though it is not entirely novel.

Weaknesses

### Novelty - The main limitation of the paper is the lack of novelty. The paper builds on DreamFusion and then adopts existing methods for articulated NeRF. It replaces the vanilla stableDiffussion model with ControlNet for 3D consistent SDS loss. Overall, the proposed approach is a straightforward combination of existing methods and it is hard for me to identify any novel technical contribution of the paper. ### Baselines - A simple baseline is missing. Similar to AvatarClip, we can extract a static mesh using MarchingCube and then rig it with SMPL. This will make it animatable. Since the paper's main contribution is animatable avatars, I believe having this baseline is important to properly validate the contributions of the paper. ### Visual quality - The quality of the generated avatars is also limited even though they are all well-known characters.

Questions

### Implementation Details: - Would be nice to provide further implementation details. What is the batch size, learning rate, guidance value, etc.? - How is the background learned during training, what are light settings, etc? ### Ablation Studies: - What is the impact of the density-weighted network? Existing methods for neural avatars (AniSDF, HumanNeRF, etc.) use different variants to accommodate non-rigid deformations. How does the performance compare with those? - What happens if random body poses from humans prior are not used during training? Would the DWN module still work? Since it operates in the canonical space, why random body poses are required in the first place?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper does not discuss the limitations of the proposed approach. Overall, the visual quality of the generated avatars is quite limited and there are severe artifacts in the animations (e.g., on the arms of Woody in support). The time required to generate an avatar is also a limitation of the proposed method and should be discussed. The qualitative results are mostly provided for the cartoonish characters which are well known. It would have been nice to see more creative avatars e.g., a doctor with a Woody's Hat, etc.

Summary

This paper proposes a new method for text-to-3D avatar generation. The proposed pipeline has two stages. The first stage generates a static avatar while the second stage learns the deformation properties of the avatar for animation. The authors propose 3D-consistent occlusion-aware score distillation sampling which seems to improve the generation quality over previous methods with standard score distillation sampling. The paper further includes the results of animation and interaction of the generated avatars.

Strengths

**Method:** The proposed 3D-consistent Score Distillation Sampling and occlusion culling is reasonable and technically sound. They seem to be effective in the provided ablation study. **Experiment:** The comparison with Sota is fair and complete to the best of my knowledge. The generated static avatars are in good quality. **Presentation:**. The paper is well-structured and written. I find it easy to follow and understand.

Weaknesses

1. I have some doubts regarding the relationship to the existing work in Table 1. DreamAvatar is marked as non-animatable, however, the original DreamAvatar paper does show reposing results in Fig 4 in their paper. This seems to contradict the claim in this paper. And regarding the interaction between avatars/objects, shouldn't this be possible with all methods as it is basically composited volume rendering if I understand correctly? 3. In L34, the authors mention that realistic animation involves changing texture and shape in different poses. However, in the qualitative results in the videos, I did not see such pose-dependent changes but only articulation with LBS. It would be helpful to visualize the pose-dependent changes in the canonical space to understand if the model indeed learns the pose-dependent effects. 4. In video 00:00-00:01, on the leftmost sequence, Woody’s upper legs disappear when they are crossing. Why do such artifacts happen? I am on the negative side currently. The main contribution of this paper w.r.t previous works seems to be making 3D text-to-avatars "animatable" and "interactive" for the first time (L49, table 1). However, regarding "animatable", previous work DreamAvatar does demonstrate avatars in new poses, and it's unclear why DreamAvatar is not "animatable". Also, the animation quality in this paper is still not satisfactory - there are artifacts (legs disappearing) and no obvious pose-dependent effects. Regarding "interactive", it seems that this is achieved simply by rendering two avatars in a composite way, which is straightforward and can be done with all previous methods. I hope the authors to clarify the contributions of this paper w.r.t previous methods.

Questions

1. Why is DreamAvatar not regarded as non-animatable? 2. How does the learned pose-dependent shape and texture change look in the canonical space? 3. What causes the disappearing leg artifacts in the video?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

There is no limitation discussed in the paper.

Summary

The work proposes a method for generating 3D skeleton animatable characters by distilling a latent diffusion model. It uses Control Net to add additional key point map conditioning to the diffusion process, improving the granularity of pose control. It uses DreamFusion to distill a NeRF model given a text prompt and the skeleton conditioning in a fixed A-pose. The model is then refined by sampling additional poses from the SIMPL model used as a skeleton prior. The model is shown to be able to generate animatable models from diverse text prompts that does not require re-training for new novel poses.

Strengths

The method proposes a stable training regime, starting with retraining using a SIMPLE mask, followed by single pose and fine-tuning on multiples poses. The addition of depth culling also helps remove artifacts seen in prior methods.

Weaknesses

- The work's main claim of animatability needs further evidence. The results show animation only for non-complex characters; no characters with long skirts or hair are animated, only reconstructed. For the samples that are shown, the animation quality is not a big departure from LBS deformation. Furthermore, there is a lack of diversity in aspects of pose that are not included in the skeleton model, for example hands are blurring and biased to a specific pose. - The ability of the method to capture interactions appears limited to blending in 3D, where complex interactions between parts are not modeled, presumably due to the independent training. For example, the hands are not clasped in the Waltz Fig. 3 - The generated models exhibit unnatural body proportions due to the mismatch between the skeleton conditioning and the text conditioning. For example, Michael Jordan (a basketball player) and Lionel Messi (a football player) seem to have the same body proportions in Fig 1. of the Supplementary.

Questions

- L22. what is the resolution of the model for rendering at 3s. - L228. Does joint training affect generalization performance? A comparison would be useful here. - How consistent is quality of results? Results showing reconstructions using different initial noise values would demonstrate diversity of generations. - L272. Is inverse LBS just the proposed method without the additional MLP for d'? Clarification would be useful. - Eqn (5). The motivation of this formula is not apparent from the text. Why would Sigmoid(d') not work similarly?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Authors have not addressed limitations or societal impact. It is suggested that the authors outline weaknesses mentioned above and suggest remediation strategies as future work. Social impact follows prior works such as DreamFusion.

Summary

This work proposes a method for text-driven human avatar generation. It combines animatable human nerf and diffusion model to implement avatar generation and animation. Extensive experiments demonstrate that its performance outperforms existing works. Also, this work supports avatar-avatar, avatar-object, and avatar-scene interactions.

Strengths

1. This is among the first diffusion-based works that can generate animatable 3D avatars, which also support the interactions between avatars and scenes/objects. 2. The occlusion culling method is well-motivated and it can address the multi-face issue effectively together with the carefully selected text prompt for viewpoints. 3. Experiments show SOTA performance for text-driven avatar generation and animation.

Weaknesses

1. This work is a little bit overclaimed (line 49-51) because there exist several prior works which can generate animatable 3D avatars with complex shapes and appearances, such as [1,2]. 2. The authors claim that DreamWaltz is able to make a scene with diverse interactions across avatars, objects, and scenes. However, it is difficult for me to evaluate whether this point is technically challenging because there is no specific design or module to enable these interactions in the proposed framework. Please explain more about this point because it is the main contribution of this work as it is mentioned in the title. 3. The proposed framework is similar to AvatarCraft, the only difference is the SMPL inverse skinning and ControlNet.Please explain more about the differences between DreamWaltz and AvatarCraft. 4. It is confusing to classify AvatarCraft as not animatable (Table 1) because AvatarCraft can also deform the canonical avatar to different target poses in the observation space. I believe by using SMPL, DreamWaltz can only change the pose parameters to implement avatar animation, which is quite similar to AvatarCraft. 5. The authors might use the term 'generalizable NeRF' carefully because when we use 'generalizable NeRF', it usually refers to the NeRF model that is not overfitted to one 3D scene but can generalize to any inputs instead. It is highly recommended to think about this term again to avoid confusion or ambiguity because the NeRF model which can be deformed to any target pose is usually named as deformable/animatable instead of generalizable. 6. I think this work is technically solid and has good performance, but there exists large improvement room for the current manuscript. [1] EVA3D: Compositional 3D Human Generation from 2D Image Collections. [2] Avatargen: a 3d generative model for animatable human avatars.

Questions

How to implement avatar-object and avatar-scene interaction? Please introduce more details for these two applications.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Not addressed. There is no discussion of broader societal impacts.