Response to updated reviewer comment
**Part1 response to updated reviewer comment:**
>Reviewer ts3J: "The authors' response addresses my first reject reason, which is appreciated. However, after carefully checking the code, I maintain my reservation regarding the heavy reliance on GLIGEN and ModelScopeT2V, which dominates the described method.”
To begin with, we would like to re-state that in addition to the first rejection reason, the third and the fourth rejection reason were already covered in our first rebuttal response.
We respectfully disagree with the assertion that our primary contribution relies heavily on GLIGEN and ModelScopeT2V. As we mentioned in the rebuttal, our paper contribution consists of three parts: (1) LLM for video plan generation, (2) Layout-guided video generation with consistency control, as well as (3) Novel evaluation metrics and datasets for measuring layout and consistency accuracy for (multi-scene) video generation, as detailed below:
(1) Firstly, we would like to emphasize that a substantial part of our work focuses on introducing the use of LLMs to generate complex, useful plans for multi-scene video generation. We start from the motivation/observation that LLMs (GPT-4) can generate bounding box movements with physical world understanding (Fig. 12), then design a multi-component video plan (Sec. 3.1) via in-context learning (Fig. 7 and Fig. 8). Finally, we conducted thorough analysis of the effect of different LLMs (Table 9), different layout control methods (Table 11), as well as error analysis (Table 8). We believe our paper has covered the analysis of LLMs in a relatively complete way. If you have any other questions or concerns, we are more than willing for open discussion.
(2) Secondly, we would like to highlight that our method is agnostic to the backbone model. To make our pipeline work, we need to build on one specific video generation model (e.g., ModelScopeT2V, the best open-source T2V model at the time). However, it's easy to adapt our framework to future video generation backbones. In addition, adopting a gated self-attention module from GLIGEN is only one part of our architecture design. We also made a non-trivial effort to enhance it with text+image embedding conditioning, proposed sharable embedding for multi-scene consistency control, and creatively trained the video generation module in a parameter-efficient way with only images (Sec. 3.2). We also did extensive ablation studies of our model architecture design (Table 4, 7, 10, 12). Therefore, stating that our method "relies heavily on GLIGEN and ModelScopeT2V'' is not well-supported.
(3) Thirdly, we provide comprehensive qualitative + quantitative experiments including novel evaluation metrics (movement direction accuracy / entity consistency) / new datasets (ActionBench-Direction/Coref-SV) for video generation, which could be beneficial for future works on layout-guided and multi-scene video generation.
----------------------------
**Part2 response to updated reviewer comment:**
>Reviewer ts3J: “Moreover, the emphasis on video rather than LM suggests that this work falls short of the expectations set by the CoLM standards."
**We strongly believe that our novel framework leveraging LMs for video generation aligns well with CoLM’s interests.** Specifically, our work addresses the topic of “17. LMs on diverse modalities and novel applications,” as outlined in the CoLM call for papers (https://colmweb.org/cfp.html).
```17. LMs on diverse modalities and novel applications: visual LMs, code LMs, math LMs, and so forth, with extra encouragement for less studied modalities or applications such as chemistry, medicine, education, database, and beyond.```
Moreover, the CoLM homepage (https://colmweb.org/index.html) mentions:
```CoLM is an academic venue focused on the study of language modeling, broadly defined, with the goal of creating a community of researchers with expertise in different disciplines, focused on understanding, improving, and critiquing the development of LM technology.```
**These underline the relevance of our contribution to the CoLM community.**
**We kindly ask for a reconsideration of the review and rating in light of these points.**