Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator

Summary

This paper presents a novel zero-shot text-video generation method that leverages an LLM as a director to generate per-frame prompts fed into a pre-trained text-image model such as Stable Diffusion. In order to retain temporal coherence, the authors propose (1) a noise distribution consisting of an interpolation of global and local noise, and (2) attend to prior frames in the self-attention layers during the generation process. In addition, they propose a dual interpolation method to generate higher fps video.

Strengths

- The paper is well written and easy to understand - The presented method is novel and interesting, and outperforms SOTA zero-shot text-video methods, as well as SOTA trained video generation methods in general frame fidelity and visual semantics, while requiring not text-video training itself - This paper presents an interesting direction in leveraging an LLM’s understanding of the world through language for video generation

Weaknesses

- Quantitative results / evaluation are a little weak, given the relatively small sample size (20 prompts, 4 videos each) - Results of ablations are shown qualitatively, and could be improved more through more rigorous quantitative evaluations

Questions

- How is $\lambda$ chosen during generation? Is it fixed, or does it change over the video timestep? - How does this method behave in scenes with more content? (e.g. more people, more animals). Can the LLM directory + SD model distinguish specifications of motions between two humans doing different things? - How about scenes with larger shifts in motion (e.g. a camera panning to the left)? As in this case frame 0 (concatenated to the self-attention) would be less useful. - Or more complex motion that may be difficult to describe in words (e.g. each step in a dance)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors discuss limitations

Summary

This paper proposes a novel approach to achieve Text-to-Video generation through Zero-Shot learning using LLMs and a diffusion-based image generation model. By using LLMs to generate detailed and varied descriptions for each frame of the video, the proposed method captures the changes in the video frames. The generated video frames are then constrained by self-attention module in SD and input noise to align with the input text and maintain a certain level of coherence. This paper also proposes a zero-shot frame interpolation algorithm based on a diffusion-based image generation model. While the current results may not be very satisfactory, this approach is a low-cost and Zero-Shot learning method that does not require additional training and can directly utilize existing SD models.

Strengths

1.The entire process is Zero-Shot learning, without the need for any additional training, making the method low-cost. Moreover, compared to existing zero-shot video generation algorithms, the proposed approach has achieved significant improvements in performance. 2.This algorithm effectively combines two pre-trained models that typically require significant computational resources for training and easy to follow. 3.Since there is no fine-tuning or post-pretraining of the existing powerful image generation model, the proposed method preserves the ability of the model to generate high-quality images to the maximum extent. Compared to models trained on video data, the image quality of the generated frames is better.

Weaknesses

1.Lacking a comparison of the quality of video generation guided by different large language models. Only ChatGPT was used in the experiments. 2.The computational and time overhead during inference may increase when using LLMs. 3.The proposed frame interpolation algorithm for video frames was not compared numerically with other state-of-the-art algorithms on the test set in the article. 4.The ablation study only provides qualitative analysis and does not include quantitative comparisons.

Questions

1.What is the test set in Table 1? I did not find any relevant explanation in the paper. 2.I didn't understand Figure 5, and what is the variable 𝜏* referring to?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1.The proposed pipline was not trained on video data, so compared to methods trained on videos, it is still weaker in terms of temporal consistency of the generated video frames. 2.The proposed method is largely limited by the high-order reasoning capability of LLMs for video imagination and the image generation capability of the diffusion-based image generation model.

Summary

In this work, the authors propose a principled pipeline for text to video generation. Specifically, three techniques are proposed: (1) a LLM based per-frame prompt generation, so that the motion/dynamics of each frame can be better specified. (2) a noise joint sampling schedule, and a step-aware attention shift is proposed to enhance temporal consistency. (3) an interpolation module to generate longer videos. Extensive experiments and results have demonstrated the effectiveness of the proposed work.

Strengths

1. The pipeline is technically sound and generally makes sense to me. It is straightforward to use the strong prior from pretrained LLM to give more temporal/motion/appearance information to each frame, to facilitate video generation. The design of noise scheme, sampling, and attention shift is technically sound. 2. The exposition of this paper is very good and clear to me.

Weaknesses

1. In my honest opinion, directing comparing this work w/ many baselines is kind of unfair, since this involves strong prior in LLM, and this is the key factor of improvement in video generation. I strongly recommend the authors to compare w/ some similar text2video pipelines that also factorize the unified single prompt condition into per-frame motion-aware conditions. For example, first generate a sequence of optical flow maps, then takes them as conditions to generate the video. 2. Since I'm not the expert in NLP domain, I'm concerned about the ability of LLM to generate very reasonable or temporally-coherent per-frame prompts. I'm also curious about the feature trajectory of such text features. Say, you generate the per-frame prompt of p1, p2, p3, ..., pn, and use the text encoder to get text features f1, f2, ..., fn for cross-attention. Do they form a smooth trajectory in the high-dimensional text feature space? Do the authors think that, a smooth feature transition among frames can guarantee a smooth video? If so, a naive idea for improvement of this work is to add some regularizations to constrain the smoothness of text features across frames.

Questions

elaborated in the above sections.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Summary

The paper proposes a zero-shot text-to-video generating pipeline called Free-Bloom, which first use LLM to generate prompt sequence decribing frames in a video, then generate frames according the prompts. To enhancing coherence, the authors proposed joint noise sampling, step-aware attention shift and dual-path interpolation. The authors compare their method with other video generator quantitatively (Clip metrics, user study) and qualitatively.

Strengths

1. explore zero-shot text-to-video generation LLM director, which leverage the story generation ability of LLM to generate semantic meaningful frame sequences. 2. Good per-frame quality. 3. tried several methods to enhancing coherence of zero-shot video generation.

Weaknesses

1. insufficient ablation study. a. there's only two qualitative cases in ablation study, which is not convincing enough. More cases and quantitative results (e.g. CLIP metrics) should be provided. b. given that joint noise sampling, step-aware attention shift and dual-path interpolation are universal technics for zero-shot video generation, experiment result of combining these technics and past zero-shot methods (e.g. text2video-zero) is needed. 2. The coherence is not good enough. e.g. sudden changes in the color / indentity can be observed. There is a fatal flaw in technical design: authors set m(t) to 1 and discard attention shift when t is small, which may lead to strong incoherence. 3. Insufficient clarification of experiments, e.g. prompts to LLM, hyperparameters.

Questions

1. Please provide more clarification of experiments, including prompt engineering to get frame description, hyperparamters (such as \tau, \tau*) 2. The coherence is not good, but the idea of leveraging LLM is interesting. Is it possible to combine LLM director with other existing pretrained text-to-video methods (e.g. make-a-video) to enhance conherence?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

see weakness and questions.

Thanks for the detailed response

I thank the authors for the detailed response, which has addressed most of my concerns! In terms of text-flow-video papers, maybe they are some concurrent NeurIPS submissions, so it's unfair to ask the authors to compare w/ them. I apologize for asking about this in the initial review. Actually, all the NeurIPS submissions in my batch of 5 papers are about DM-based text-to-video generation, and imho, this paper's technical soundness, exposition, and contribution could be the best. So I'm raising my score to 7 and vote for the acceptance of this paper.

Reply to Reviewer 6FyB: Thank you!

We thank the reviewer for the quick reply and consideration! It is heartening to receive this positive evaluation. Your detailed review and feedback are valuable to us in further shaping our work. Once again, we express our gratitude for your time and appreciation!