Compositional 3D-aware Video Generation with LLM Director

Summary

This paper presents an LLM-involved three-stage pipeline for text-guided compositional 3D-aware video generation. In the first stage, an LLM is employed as director to decompose input textual prompts into sub-prompts including scene, object and motion. Subsequently, it leverages multi-modal LLM to make an initial estimation about scales and trajectories for each object. Moreover, 2D diffusion priors are further leveraged to refine the 3D generation results with SDS loss. Extensive experiments demonstrate that the effectiveness of the proposed method in 3D-aware high-fidelity video generation.

Strengths

1.The idea of decompose the generation task into scene, object and its motion in a more organized way with LLM is interesting. 2.The method of generating trajectories following a step-by-step manner is a reasonable solution and is validated in the experiments. 3.Using SDS loss to distill generative priors from pretrained LDM/SD models helps refine the quality of generation and is validated in the ablation study. 4.With LLM as director and 3D as structural representation, the proposed method is able to generate 3D-aware videos with diverse motion and high visual quality.

Weaknesses

1.How to choose pretrained expert models? How would those models influence generation performance? 2.Since the proposed method adopts the divide-and-conquer strategy, I wonder how it perform in case of complex scene with multiple objects, especially with interaction among objects.

Questions

1.Rendering 2D video from 3D representation is real-time as said in the paper, what about the time spent on 3D representation generation and trajectory generation ? 2.How is the influence of selection of pretrained expert models and open-source LLM/MLLM as director?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, limitations have been discussed in the paper.

Summary

This paper proposes a method to synthesize dynamic 3D videos, with moving objects and camera. It treats the tasks compositionally, generating the (static) background and foreground figures separately. The generation process is orchestrated by an LLM, which provides prompts to separate models that specialize in different scene components (and which are pretrained); these components each use gaussian spats allowing assembly into a single dynamic 4D scene. This compositionality also enables controllable generation, where certain scene elements are replaced depending on user input. The method is demonstrated on several text prompts, and shown to out-perform three baselines.

Strengths

- The idea of treating 3D/4D video generation as a compositional task is elegant, and it is sensible to leverage existing strong domain-specific models; it is also a nice idea to use an LLM here to automatically determine a suitable sequence of domain models to apply, and how to combine them. - The proposed pipeline is novel, and fairly natural for the task. The choice of stages/components is clearly motivated. - The method successfully generates videos from at least two text prompts, showing somewhat plausible motion. Qualitative results from two prompts show significantly better visual quality than the selected baselines (VideoCrafter, Comp4D, 4Dfy) - Quantitative results based on CLIP score (adherence of frames to prompt) and Q-Align again exceed the baselines - As well as text-conditioned generation, the method also supports certain other kinds of controllability. Since the scene representation is compositional, foreground humanoids can be replaced by others, motion can be modified, and the background can be replaced. This affords a degree of precise control that is missing from 'monolithic' video generation models - Ablation experiments were conducted, removing three components, aiming to establish their importance in the overall pipeline - The writing is generally clear (modulo a few grammar issues); the paper is well organized.

Weaknesses

- Very few qualitative examples are given (and presumably these were cherry-picked rather than random). In particular, only two text-conditioned generations are shown (same in paper and supplementary). This makes it difficult for the reader to judge the visual quality of the model outputs, which is vital for such a task - Even in the given two examples, prompt adherence is poor, with the "skeleton man" missing, incorrect positioning ("in front" of the stage vs at the front; "in" a cabin vs outside), and unrealistic lighting (no shadowing). This is problematic given that object compositionality being guided by the LLM is claimed as a key contribution of the work - It is unclear how many videos were used for the quantitative evaluation, nor where the set of prompts was drawn from. This makes it hard to judge the significance of these results. - The ablation experiments are on a prompt and qualitative only. This means they are statistically meaningless, and the benefits of the different components need to be demonstrated more rigorously. - The exact prompting strategy for the LLM is unclear, in particular the initial stage of creating the sub-tasks, and the creation of the scale/trajectory estimation prompt. - The method is limited by the choice of 'experts' that synthesize parts of the scene (currently static background from LucidDreamer, humanoids from HumanGaussian, and humanoid motion from Motion-X). While 'delegating' generation subtasks is a neat idea, it seems that significant engineering work is required to incorporate each 'expert', and there is not a clear path to adding e.g. other dynamic object types such as quadruped animals.

Questions

- How many prompts were in the evaluation set? How were these selected? - What are the quantitative results from the ablation study?

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

There is adequate discussion of limitations. There is an exceedingly brief discussion of broader impacts, borderline adequate for this task.

Summary

In this work, the authors propose a framework for 3D-aware video generation using guidance from LLM. Specifically, this work follows previous studies on LLM for video generation that takes language model as a director to do below sub-tasks: 1) expand and decompose the prompt into different aspects, and then use off-the-shelf expert models to generate objects/motions/scenes. 2) plan the trajectory and other scene configurations. 3) put all of them together and update with SDS loss. Experiments are conducted to verify the effectiveness of this work.

Strengths

1) The focused setting is novel, which enables LLM-guided text-to-video generation with 3D awareness. 2) The proposed pipeline generally makes sense to me.

Weaknesses

1) The novelty of this work is limited. It seems like to be the combination of 3D scene generation + 3D avatar generation + motion generation + LLM + SDS. It would be subjected to the performance upper bound set by each expert model, and combining them altogether would make the quality even worse. 2) In TC4D, the trajectory can also be generated by LLMs, which should not be a drawback of that work.

Questions

How do the authors handle the situation where scene configurations are not properly

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations are discussed.

Summary

The paper presents a pipeline for 3D-aware video generation by composing scenes, objects, and motions. One key idea is to use existing LLM to provide coarse guidance on the scale and trajectory of objects, and then refine the coarse rendering with SDS. The method is compared with multiple methods.

Strengths

The idea is explainable as it composes the scene, objects, and motions in an explicit manner. So potentially it is easy to improve the pipeline by building better modules for each component. The paper shows both quantitative results and qualitative examples. The paper is overall easy to read.

Weaknesses

$\textbf{Method}$ 1. It would be great to clarify what are considered to be the main contributions of the paper. The pipeline uses multiple existing modules for 3D generation, and then using LLM to generate a series of bounding boxes for the guidance of the rendering. The idea of using LLM to generate a trajectory does not look novel enough given the existing volume of literature. 2. For the refinement stage, is the scale and 3D location refined for each image individually? If so, how does the scale and location maintain temporal consistency? 3. How does the motion be compatible to the scene? Suppose it's a rough terrain or mountain area, the motion has to be adjusted? 4. To generate the bounding box trajectory, is it correct that a single image is used as input, which means no 3D information is used for LLM? If the 3D scene is existing, do you think adding 3D information and generating 3D locations could be a better option rather than 2D bounding boxes? $\textbf{Experiments}$ 1. One concern is that the overall quality of the human actors is still limited. Since the resolution is not high enough, it is hard to tell whether the method is harmonizing the image well given the examples presented. 2. It is also hard to see the 3D viewpoints changes given the examples, it seems like for most examples, the viewpoints are not changing much for the sequence of renderings. I imagine identity-preserving character inpainting methods can show similar examples? Maybe show more examples demonstrating the uniqueness of the proposed method. 3. The baselines do not include the strong motion prior as the proposed method so I am not sure if this is fair. How does the method compare to methods with explicit 3D priors [1]? [1] Image Sculpting: Precise Object Editing with 3D Geometry Control. CVPR 2024.

Questions

I think the paper tackles an interesting problem but the effectiveness of the solution remains to be justified.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Response to rebuttal

I thank the authors for their detailed responses. In particular the additional qualitative results are appreciated, as well as the ablations. While I still have concerns about quality (particularly in terms of prompt adherence), the rebuttal largely addresses my concerns.

response

I appreciate the efforts of authors in responding my comments. My concerns have been partially addressed, but the influence of expert models and LLMs/MLLMs are not investigated sufficiently. After reading other reviewers' comments, I also think the quality of the generation should be further improved. Hence I would change my score to 5 borderline accept.