YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals

Summary

1. This paper proposes to generate animals with pose control. 2. The pose control is achieved by controlnet trained on pose-image pair data. 3. The pose can be generated with LLM during inference.

Strengths

1. The pipeline consists many parts and some efforts are put to implement the whole pipeline. 2. Experiments show robustness in animal generation compared to previous optimization-based methods. 3. Introducing LLM into 3D generation is novel.

Weaknesses

1. The description of how you implement LLM to generate poses in Sec. 3.2 is too vague: * Which LLM do you use? Did you fine-tuned the LLM on pose data? * How do you represent the pose in natural language to feed into the LLM? * Do you represent the pose with discrete tokens or continuous real number? * How do you implement the 3 agents (obsever, modifier, finder)? Many details above are lack. These details are very important for the readers to understant how the method is implemented and evaluate the technical soundness of this work. I think the authors may provide these details in rebuttal and in the further revision of this manuscript. Currently, I suggest a reject owing to the presentation. 2. No results on the failure cases are reported. No details about the success rate of generation are reported. I suggest the authors to report a metric of the success rate of generation. Sometimes the generation results consist of much artifact and are considered as failure cases. As for success rate, I suggest the authors to present the portion of generation results that are not failure cases.

Questions

See the weakness above.

Rating

5

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

See the weakness above.

Summary

The paper presents YouDream, a framework for text-to-3D animal generation. The two keys to their methods are (1) a tetra-pose ControlNet that synthesizes animals given a text prompt and 2D tetrapod poses, and (2) a multi-agent LLM system that modifies 3D keypoint templates to generate different animal poses. Using the generated 3D animal poses as guidance, and the ControlNet for Score Distillation Sampling (SDS), YouDream can optimize Neural Radiance Fields (NeRFs) with better geometry and appearances, alleviating the multi-faces problem common in previous text-to-3D approach. Qualitative comparisons and user studies in the paper suggest that YouDream achieves more favorable results compared to previous text-to-3D methods for 3D animal generation.

Strengths

The presented approach makes sense, and shows promising results. In particular, it has the following strengths: - An automatic system that produces 3D poses/skeletons for various animals. This is a very creative way to exploit the LLM. - The framework can produce 3D animal assets with more accurate 3D geometry, reducing the multi-face problem that plagues many SDS-based approaches. - YouDream offers controllability over the animal poses, making it a potentially useful tool for 3D artists. In summary, the paper presents a promising framework that tackles an interesting problem (3D animal generation) that can benefit 3D artists and beyond, and the novelty lies in how YouDream exploits the LLM to handle diverse, varying animal skeletons.

Weaknesses

YouDream has some weaknesses - Color saturation artifacts. This is a common problem among SDS-based approaches and could be fixed using implementation tricks from other work. - While greatly alleviating the multi-face problem, we can still observe artifacts (e.g., multiple pairs of ears in Supp. raccoon_standing), and extra limbs (Figure 8, tiger plush toy). The problem could be mitigated with a more detailed skeleton (i.e., more landmarks/keypoints). - Controllable, but not editable – YouDream has to re-run the optimization process to create the same 3D animal in a different pose. - Similar to many other text-to-3D methods, no quantitative/objective evaluations available. While the presented results look pretty good, we cannot assess/quantify the improvement over previous methods. This, however, is not a problem unique to YouDream.

Questions

Please clarify the following questions if possible: - L7-8: It might be more appropriate to say “generates 3D animals with controllable poses”. As shown in Figure 1, Figure 4, and Figure 15, other methods can also generate 3D animals, sometimes with even more appealing textual qualities (i.e., Figure 15 MVDream Giraffe). - There are flickerings in some of the supplementary videos across views. Since the underlying representation is a NeRF, I expect the 3D results to be view-consistent. It would be great if you could clarify if the flickerings are caused by compression or other issues. - L169-172 mentioned that TetraPose ControlNet works well for out-of-domain animals, which is somewhat backed by the llama octopus example (Figure 1). It would be interesting to see more results on non-tetrapods, such as snakes, fishes, insects, etc. This is a bit out of the scope of this manuscript, but it would be great to see to what extent the proposed multi-agent 3D pose generator and TetraPose ControlNet work.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are addressed appropriately in the paper. Please answer/clarify the above-raised questions/weaknesses if possible.

Summary

This work proposes a method to generate 3D models with pose control. It mainly contains two steps: 3D pose generation with LLM agents and text-to-3D generation with pose-conditioned ControlNet and viewpoint sampling. From the experiments, the method can generate novel 3D animals with controlled poses, while previous methods without pose conditions can not.

Strengths

1. The method enables the generation of 3D tetrapod animals with diverse poses. 2. The method can generate novel animals that are out of the domain of training data. 3. The pose-guided generation enables more fine-grained pose control for the text-to-3D generation.

Weaknesses

1. The work allows more explicit pose control for generating tetrapod animals. However, original text-to-3D models can achieve this using only text, without limitations on object classes or the need for human labor to design poses. 2. The poses appear to be user-designed, but I still doubt the difficulty of creating novel poses with the given library, like it's hard to create the tiger walking sequences (Fig. 21) given the library, are the poses generated from Multi-agent LLM or designed manually. 3. The effectiveness of the LLM-based pose editor seems limited; how can a 2D-based model (LLM) effectively reason about the 3D poses? The given initial poses in Fig.3 are already plausible. What if more noisy poses are given? 4. The generated model is of bad quality and not comparable with MVDream, although it has a 3D GT. I doubt whether adding the explicit pose control will make the mesh quality worse.

Questions

What are the finder, observer, and modifier composed of, respectively? What modifications can the modifier make to the given 3D poses? For the text prompt “A giraffe with dragon wings,” are the poses automatically generated by the modifier?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Summary

The paper proposes a novel technique called YouDream for text-guided animal generation. Specifically, the authors first propose a multi-agent LLM that's capable of generating a 3D pose of the text-described animal. Secondly, YouDream includes a TetraPose ControlNet to generate the images based on the projection of the 3D pose. Last but not least, the authors design a tool for generating 3D rough shapes based on the 3D pose, which will serve as the initialization for the generation process. Extensive experiments, including qualitative, quantitative, and ablation studies, have been included to demonstrate the effectiveness of the network.

Strengths

The strengths of the proposed paper can be summarized as: + The paper proposes a multi-agent LLM to generate 3D pose of animals, which is novel and interesting for 3D generation. + The results are impressive, showcasing better performance than existing techniques.

Weaknesses

The weakness of the proposed paper can be summarized as: - Missing comparisons with GaussianDreamer, LucidDreamer, etc, which are more recent works than 3DFuse, Fantasia3D, and HiFA. Meanwhile, it would also be better to compare with DreamFusion, Magic3D, and prolificdreamer to further demonstrate. - The ablation in Fig. 6 is not positive. Actually: it seems that the initial shape provides more contributions, while the pose control doesn't affect much. - It would also be good to show quantitative evaluations using CLIP-based metrics. - The methods can only process 16 categories of animals. Although it's a limitation of current datasets, it will still hurt the method's performance and bring the question of whether we need the technique specifically designed for these animals.

Questions

Please see weakness listed above.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitation has been discussed in the Appendix.

Thanks for the detailed responses

Thanks for the comprehensive responses. My concerns are all addressed, and I would love to know other reviewers' opinions after reading the rebuttal. For now, my feedback regarding YouDream remains positive.