VideoTetris: Towards Compositional Text-to-Video Generation

Summary

This paper proposes VideoTetris, a novel framework for compositional text-to-video generation. It addresses the limitations of existing methods in handling complex scenes with multiple objects and dynamic changes. VideoTetris achieves this through several key innovations: I) Spatio-Temporal Compositional Diffusion: Manipulates the cross-attention of denoising networks to synthesize videos that follow complex instructions. II) Dynamic-Aware Video Data Processing: Filters and recaptions video-text pairs to enhance consistency in auto-regressive long video generation. III) Consistency Regularization with Reference Frame Attention: Maintains coherence in multi-object generation by aligning object features across frames. Extensive experiments demonstrate that VideoTetris significantly outperforms state-of-the-art methods in both short and long video generation tasks, showcasing its ability to generate high-quality, coherent, and compositional videos.

Strengths

1. The paper is easy to follow. 2. This paper introduces a new approach for compositional text-to-video generation, addressing the limitations of existing methods in handling complex scenes and dynamic changes. 3. Extensive experiments demonstrating the superior performance of VideoTetris compared to state-of-the-art methods in both short and long video generation tasks.

Weaknesses

1. This paradigm introduces more computational cost by the decomposition of input text using LLMs and the computation of cross-attention for multiple sub-objects and frames. These operations require more computation, especially in scenarios with numerous objects or a high number of frames, potentially impacting the efficiency of the overall video generation process. Besides, the use of ControlNet for auto-regressive generation leads to a high computational cost, which may limit the practical applicability of the method. 2. The Reference Frame Attention module relies on the assumption that object features remain consistent across frames. However, this may not always hold true, especially for dynamic scenes with fast movements or significant changes in lighting. Investigating more robust consistency regularization methods that can handle these challenges would be beneficial. 3. The generated videos exhibit relatively subtle variations in content, which could result in static or less dynamic scenes. This may limit their practical applicability.

Questions

N/A. See weakness for more details.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed both limitations and potential negative social impacts.

Summary

The paper presents "VideoTetris," a new framework designed to improve text-to-video generation in complex scenarios with dynamic changes and multiple objects. It introduces spatio-temporal compositional diffusion techniques for better alignment with textual semantics and integrates a dynamic-aware data processing and consistency regularization to enhance video consistency. The results from extensive experiments demonstrate significant qualitative and quantitative improvements in T2V generation.

Strengths

1. The paper is well-written and easy to understand. 2. The motivations for spatio-temporal compositional diffusion and Dynamic-Aware Video Data Processing are clearly explained, with supportive experimental evidence provided. 3. The experiments are sufficiently thorough, demonstrating that VideoTetris achieves impressive results in both short and long video production scenarios.

Weaknesses

1.Although Spatio-Temporal Compositional Diffusion directly adjusts cross-attention, it still segments videos into different shots, each with a specific layout, suggesting it's essentially a layout-based method. This approach contradicts the motivation outlined in section 3.1. 2.The paper presents limited technical innovations, focusing more on engineering implementation. In Spatio-Temporal Compositional Diffusion, it uses LLMs to pre-process prompts and region masks, which are then sequentially applied during generation. Meanwhile, Dynamic-Aware Video Data Processing enhances data preprocessing for higher quality.

Questions

How is continuity ensured between different shots?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The manuscript includes discussions on limitations and social impacts.

Summary

The paper proposes VideoTetris, a novel framework for compositional T2V generation. It introduces Spatio-Temporal Compositional Diffusion method for handling scenes with multiple objects and by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, authors propose a novel Dynamic-Aware Data Processing pipeline to enhance auto-regressive long video generation and a consistency regularization method with Reference Frame Attention that maintains coherence in multi-object generation.

Strengths

1. Clarity: The paper exhibits well-written text that is clear and easy to follow. This quality contributes to the overall readability and accessibility of the research. 2. Method: The proposed framework presents a powerful solution for enhancing multi-object T2V generation, while also enabling the generation of long videos. This method offers significant advancements in the field and addresses the challenges associated with generating videos containing multiple objects. 3. Resource-Friendly: Remarkably, the proposed method is resource-friendly, as it only requires 4 A800 GPUs for fine-tuning the model. This efficient utilization of resources makes the method more accessible and practical for implementation in real-world scenarios.

Weaknesses

1. Novelty 1.1. The paper lacks clarity regarding the specific novelty introduced by the Spatio-Temporal Compositional Diffusion method. As per my understanding, this method comprises two components: Localizing Subobjects with Prompt Decomposition and Spatio-Temporal Subobjects Composition. However, similar components can be found in existing literature [1, 2, 3]. It is recommended to emphasize the unique aspects and novelty of the paper to distinguish it from prior works. 1.2. The authors highlight the proposed Dynamic-Aware Video Data Processing pipeline as one of the paper's major contributions. However, it appears that this pipeline involves filtering videos with an optical flow score, which has already been proposed in Stable Video Diffusion. If this statement is accurate, the pipeline cannot be considered a contribution of the paper since it lacks novelty. 2. Experiments: 2.1. The paper's qualitative results are supported by only three examples for Short Video Generation with Single Multi-Object Prompts and two examples for Long Video Generation with Progressing Multi-Object Prompts (as shown in figures 1, 4, and 5). This limited number of examples is not statistically representative. It is recommended to provide more samples for visual comparison and conduct a user study with at least 50 samples and 15 users to ensure a more comprehensive evaluation. 2.2. For Short Video Generation with Single Multi-Object Prompts, the authors compare their method with standard T2V models that are not specifically designed for Compositional Video Generation. It is recommended to include a comparison with LVD 1, as mentioned in Section 2, to provide a more relevant baseline for evaluation. 2.3. The paper lacks ablations for the Spatio-Temporal Compositional Diffusion method. It is recommended to compare the proposed Localizing Subobjects with Prompt Decomposition pipeline with relevant prior works, such as those mentioned in [1] and [3]. Similarly, the Spatio-Temporal Subobjects Composition should be compared with the approach described in [2] to provide a more comprehensive analysis of the proposed method's effectiveness.

Questions

1. Novelty Clarification: It is crucial for the authors to provide a clear explanation of the novelty introduced in their paper compared to existing works. This clarification should highlight the unique contributions and advancements made in the proposed approach, distinguishing it from prior works in the field. 2. Baseline Comparison and Ablations: The authors should compare their method with stronger baselines specifically designed for Short Video Generation with Single Multi-Object Prompts. This comparison will provide a more comprehensive evaluation and demonstrate the superiority of their approach. Additionally, conducting ablations for this specific component of the proposed method will further enhance the understanding of its effectiveness and showcase its individual contributions. [1] Lian, Long, et al. "LLM-grounded Video Diffusion Models." *The Twelfth International Conference on Learning Representations.* [2] Chen M., Laina I., Vedaldi A. Training-Free Layout Control with Cross-Attention Guidance //2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). – IEEE, 2024. – С. 5331-5341. [3] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." *arXiv preprint arXiv:2309.15091* (2023).

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors adequately addressed the limitations.

Summary

The paper presents a novel framework designed to improve text-to-video (T2V) generation, especially for complex scenarios involving multiple objects and the composition of different objects. The proposed VideoTetris introduces spatio-temporal compositional diffusion, a dynamic-aware data processing pipeline, and a consistency regularization method to enhance the quality and coherence of generated videos. Extensive experiments demonstrate the framework's superior performance in generating both short and long videos with complex and multi-object prompts.

Strengths

- I like the idea that the model first splits a story prompt into frame-wise short prompt with object composition. This approach supports sharing global information among long synthesic video (which is usually computationally expensive and hard to achieve). - The visual quality provided in the supplementary looks great (although the video number is limited) - The experiments include the quantitative comparison with the state-of-the-arts in the tasks of short and long video generation with multi-object prompts, and also ablation study, which can provide better understanding about the performance of the proposed VideoTetris.

Weaknesses

- My major concern is that the model heavily relies on the performance of LLM. In the proposed method, LLM acts as a director to split an input story prompt into spatiotemporal prompts. I am wondering whether LLM is robust enough to generate temporal-consistent prompt with spatial-consistent composition. - Following the concern above, the supplementary only provides one example of long video generation. The authors should more detailed and more samples. For example, provide 100 story prompts and the corresponding frame-wise prompts with object composition. For what we have now, it is hard to fairly justify the model performance. - Another concern is the autoregressive strategy of long video synthesis. While the long video shows good consistency of motion and subject identity, I found the visual quality gradually decreases in the video. For example, in the sample "fig5-ours.mp4", the contour of leaves is visible at beginning; however, it gets more and more blurry until the end of the video. Is this also happens on other long synthesis videos?

Questions

- Does the proposed framework have the scalability limitations in terms of the length and complexity of the generated videos? - How does the framework perform in real-world scenarios where the input text prompts may be less structured and more diverse?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

- Scalability: as mentioned in the weakness, the model highly relies on the performance of LLM which has unstable performance and sometimes has hallucination issue. Hence, it requires manual check and limits the scalability. - Limited length of long video generation: also mentioned in the weakness, autoregressive strategy of long video synthesis limits the length of generated video. Could the author provide more or even longer video to verify this limitation?

Response to Rebuttal

Dear Authors, Thank you for your comprehensive rebuttal. I appreciate the effort you have made to address all my concerns. Based on this, I have decided to raise my score and recommend acceptance of your paper. I would also like to commend the authors for conducting additional experiments. In my opinion, these results provide valuable insights and would be a useful addition to the article. Therefore, I strongly recommend that they be included in the camera-ready version of the paper.