FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention

Summary

The paper introduces a SpectralBlend Temporal Attention (SB-TA) mechanism, which blends low-frequency and high-frequency components from attention features together to enhance consistency and realism in generating long videos. The authors tested the proposed algorithm using 25 text prompts on LaVie and VideoCrafter, demonstrating improvements in VBench metrics. Additionally, they conducted ablation studies to showcase the characteristics of different modules in the proposed method.

Strengths

1. The proposed method achieved notable improvements, and the authors conducted interesting analyses to support their findings. 2. The proposed method supports multi-prompt long video generation.

Weaknesses

1. My first concern is that 128 frames itself cannot be considered long video generation. Therefore, it would be more convincing to see how effective the authors' proposed method is on longer videos, such as generating 1-minute videos like Sora. 2. The proposed method includes several hyperparameters, such as alpha and tau. Are these parameters significant in influencing the final results? Because the authors tested only 25 prompts, it's unclear whether these hyperparameters remain suitable across a broader range of scenarios. 3. Although adding local features to long-range features can mitigate some loss of spatial details, for longer video generation with significant content variation, it remains uncertain whether these local features adequately supplement spatial details.

Questions

1. Figure 2's right panel demonstrates strong diagonal correlations in VideoCrafter at 64-frame and 128-frame intervals. How does this correlate with the findings proposed by the authors? 2. Since the focus of this paper is to test the algorithm's ability in long video generation, while VBench primarily benchmarks the quality of short video generation, the key question is how to demonstrate that the results obtained reflect the algorithm's performance in long video generation quality.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The method proposed by the authors can only generate videos up to 128 frames, which falls short of true long video generation. It has yet to be validated in generating genuinely long videos.

Summary

This paper proposes a training-free method to generate 8x longer videos based on 16-frame pre-trained video diffusion models. It observes that extended temporal attention has a negative effect on high-frequency generation. Thus, it proposes an SB-TA module to fuse global low-frequency temporal features and local high-frequency temporal features.

Strengths

1. Lines 42-43 & 107-121: The observation on SNR with spatial-temporal components is natural and reasonable. 2. Novelty: While frequency splitting and merging are common in low-level image processing tasks like super-resolution, this is the first solution for video temporal "super-resolution".

Weaknesses

1. Lines 69-76: The related work section is not reader-friendly for those unfamiliar with the field. Simply listing papers across a wide range (from GAN to diffusion) is too vague to introduce the outline of the research field. 2. Lack of comparisons and baselines: The observation analysis and experiments only include two base models: LaVie and VideoCrafter. Note that both of these video diffusion models adopt relative positional encoding (PE) in their temporal attention blocks. Other models like AnimateDiff (absolute PE), DIT-based models like Open-Sora (RoPE), and ModelScope (which adds temporal convolution, no PE) are not discussed in the paper. This suggests that the proposed method may be limited to one specific PE. 3. The visual result quality is limited and does not include more motions, despite the frame rate being 8x (16 to 128). Although the authors show multi-prompt generation ability in Sec. 5.5, a single prompt should still be enough to cover a much longer motion. (e.g., Fig.7's first prompt, "running Iron Man," is a complex motion, and it should be generated correctly as a complete motion with 128 frames sampling. Can you show this case in the rebuttal?)

Questions

1. In Lines 33-38 and Fig. 1, the authors claim that longer video generation on short-clip pre-trained models will degrade high-frequency details. However, the results of FreeLong still do not have rich details (especially in the background). 2. Did you consider the memory cost of long-video generation with one forward pass (128 frames)?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes.

Summary

The auther propose FreeLong, a training-free method for long video generation. This paper identifies that the problem with long video generation lies in the scope of attention and the distortion of high-frequency components. Based on the observation, the auther proposes a novel method to blend different frequency components of global and local video, leading to better result in long video generation.

Strengths

1. The research problem is important. The paper is well-written and easy to follow. 2. The observation and the method are reasonable. 3. Experiments results look promising.

Weaknesses

My main concern lies in the computational cost.

Questions

1) Computing the global attention map and video features seems computationally expensive, what is the computational cost compared to previous method (like freenoise) ? 2) Method like freenoise suffer from repetitive generation. Generated contents will repeat several times. Can freelong solve this problem?

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

The paper presents FreeLong, a novel training-free method for generating extended videos (128 frames) using pre-trained short video (16 frames) diffusion models. The key component is the SpectralBlend Temporal Attention (SB-TA), which fuses low-frequency global video features with high-frequency local features to ensure consistency and fidelity in long video generation. Experimental results highlight FreeLong's superior performance in video fidelity and temporal consistency over existing methods.

Strengths

1. **Comprehensive Analysis**: The paper conducts an in-depth frequency analysis to identify challenges in long video generation and supports the proposed solution with extensive experimental validation. 2. **Novel SB-TA Mechanism**: The proposed SpectralBlend Temporal Attention effectively mitigates high-frequency component degradation, ensuring consistent and high-fidelity video outputs. 3. **Training-Free Long Video Generation**: Long video generation is an important task and future direction. I am happy to see a training-free method appear in the community. The proposed FreeLong is both resource-efficient and practical, allowing for the adaptation of existing short video models to long video generation without the need for retraining. 4. **Multi-Prompt Generation Support**: The multi-prompt video generation is coherent within FreeLong, and the visual continuity and smooth transitions between different scenes are satisfactory.

Weaknesses

1. Inference time comparisons should include both single-pass and multi-pass temporal attention from previous methods to demonstrate the advantages clearly. 2. Although the video generation results are impressive, a more extensive user study is required to validate the consistency and quality of the generated videos.

Questions

Some concerns please refer to the Weaknesses. Also what about the longer duration of video generation, for example, videos longer than 128 frames?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Including examples of failure cases would provide a better understanding of the method's limitations.

Good rebuttal

Thank authors for the thorough response and the additional experiments results. I am satisfied with the rebuttal and no longer have any further concerns. Therefore, I have decided to raise my score to accept.