Summary
The paper proposed a multi-stage approach to generate long videos that encompass diverse content and multiple events, which utilizing additional guidance to steer the whole video generation process. These additional guidance includes object labels, box, and layout.
Specifically, this paper first predict all the keyframes jointly based on the input guidance. These keyframes represent the starting
frames of each video clip. Then generating the entire video by predicting the intermediate frames between keyframes, using the video clip generation model.
The proposed method outperforms state-of-the-art video generation models by 9.5% on FVD.
Strengths
1. The approach of initially predicting keyframes for different clips in a long video and then interpolating seems reasonable. Indeed, to some extent, it can address the issue of repetitive content in long videos.
2. Constrain the interaction between layout and keyframe to preserve the object coherency in the long video using layout-keyframe attention mask.
3. Achieved good performance on two public datasets.
Weaknesses
1. This work is meaningful and has demonstrated good performance on some mainstream datasets, marking a significant improvement compared to previous efforts. However, in the era of large models and datasets, video diffusion models have shown more promising performance with features such as high resolution and more reasonable motion [1][2][3]. If the authors could further expand their training dataset to enhance performance and generalization capabilities, it would make this work even more noteworthy. Of course, the reviewer understands the substantial effort required for such tasks and does not necessarily expect the authors to provide additional experiments or proofs.
2. Furthermore, the reviewer recommends that the authors provide more visual demonstrations from other domains, not limited to cooking and driving scenarios. This would offer a more comprehensive showcase of long video generation applications across various scenes.
3. Figure 4 presents a comparison of relevant quality degradation. An interesting observation is that using GT layouts appears significantly better than layouts predicted by the transformer. Layouts seem to play a crucial role in this method. The reviewer recommends that the authors provide visualizations of layouts, especially in bad cases. This can assist readers in understanding the performance of the layout component.
[1] Wang, Yaohui, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang et al. "LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models." arXiv preprint arXiv:2309.15103 (2023).
[2] He, Yingqing, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang et al. "Animate-a-story: Storytelling with retrieval-augmented video generation." arXiv preprint arXiv:2307.06940 (2023).
[3] Ho, Jonathan, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma et al. "Imagen video: High definition video generation with diffusion models." arXiv preprint arXiv:2210.02303 (2022).
Questions
1. User study. The paper conducted the study with 40 videos and 11 participants. Perhaps the authors could provide a brief introduction to the origin and data distribution of these 40 videos, along with any relevant references if they align with previous works. Additionally, clarification on the participant demographics would be helpful, such as whether they are members of the author's team or hired participants. Providing this information would enhance the overall understanding of the study.
Rating
6: marginally above the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.