Fast and Memory-Efficient Video Diffusion Using Streamlined Inference

Summary

In this paper, the authors focus on reducing the computational requirements and high peak memory usage for video diffusion models. Specifically, a train-free framework is proposed, which consists of three parts: Feature Slicer, Operator Grouping, and Step Rehash. Those three steps result in significant memory reduction and inference acceleration.

Strengths

1. The proposed framework has a great performance for reducing peak memory and accelerating the inference of video diffusion models. In particular， the peak memory of AnimateDiff can be reduced significantly from 41.7GB to 11GB, which can contribute to more practical applications. 2. The pipeline of the proposed framework is easy to understand and is well-documented.

Weaknesses

1. More base models with other frameworks of video diffusion should be involved for more comparisons. The two baselines used in experiments, i.e., SVD and AnimateDiff, are both based on T2I diffusion models and Unet backbone. Is the proposed framework suitable for other diffusion models with DiT backbone[1] or 3D diffusion, like Open-Sora Plan v1.1(can be found in Github). 2. Human-level metrics should be involved to compare the video visual quality in Table 2. The FVD and CLIP-Score may not be enough to measure the performance. Based on the cases shown in Figure 1 and 7, I prefer to agree that the original base model has better performance. [1]: Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

Questions

My main question is about the generalization of the proposed framework. Based on my understanding, most contributions of this paper are “engineering”. It is important to verify the performance on various base models, such as diffusion models with DiT backbone and 3D diffusion models. In addition, if I want to use this framework, do I need to modify a lot of content in the code?

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Summary

This paper proposes a training-free video diffusion inference acceleration method, which includes three processes: Feature Slicer, Operator Grouping, and Step Rehash. Compared to the baseline, the proposed method shows significant improvements in memory usage and inference speed.

Strengths

- The method description is straightforward and easy to understand. - The proposed method is effective, simple, and easy to implement, and it shows significant improvements. - The problem addressed by the proposed method is critical.

Weaknesses

- The experiments need to be improved. Step Rehash can greatly accelerate the inference speed, but it seems to be highly dependent on the weights of the video diffusion model. It is unclear whether the proposed method can be applied to most video diffusion models. Therefore, the authors need to compare more types of models and provide statistical information on the performance changes, such as mean and variance. - The proposed method shows a visible loss in video generation capability, as seen in Figure 7 where the owl's eyes disappear and the overall texture quality of the synthesized image decreases in the optimized version.

Questions

- From the images, it is clear that the quality of some of the synthesized images has decreased. Based on my experience, this decrease in quality may have a more severe impact on video synthesis tasks. The authors did not seem to provide video demos, so it is difficult to judge how severe the loss in performance is. - What is the relationship between the optimization method and batch size used during inference, and how do they affect the optimization speed and memory usage?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

I think this paper addresses an important problem and appears to have significant improvements. However, the degree of loss in synthesis quality is not clear, which is an area of concern for me. The loss in quality seems to be less noticeable in SVD, but more significant in AnimateDiff (e.g., the owl in Figure 7, the bus in Figure A6, and the parrot in Figure A5).

Summary

This paper presents a framework for reducing the computational demands of text-to-video diffusion models. The main idea involves dividing input features into subfeatures and processing them parallelly, thereby reducing peak memory usage during sampling. To address the increase in overall compute time caused by this partitioning, the authors propose a skip strategy that determines when and where to apply the skip operation based on feature similarities.

Strengths

1. The paper is well-written and motivated. 2. The proposed method is intuitive.

Weaknesses

While the paper is well-motivated for the important problem and proposes an intuitive and simple remedy, there are major concerns regarding the evaluation, scalability, and applicability of the proposed method, as follows: **Scalability and Applicability** - My major concern is the scalability of the proposed method. Since the method requires dividing the input features into patches, it inevitably increases the computation as the feature dimension gets larger, potentially introducing more patches to compute. While the proposed skip strategy might mitigate some of the increased computation time, it definitely harms the quality of the generated videos. - In terms of applicability, the proposed framework is only applicable to U-Net-based video diffusion models. However, recent models, such as those utilizing DiT backbones, are not addressed by this framework. **The evaluation is weak** - Lack of baselines: The authors only compare their method to naive hashing, which is the most simple baseline. To verify the effectiveness of the proposed method, please include existing method for efficient sampling that also reduce memory and sampling speed. The authors only compare with DeepCache. They should compare with more training-free baselines, even though these primarily demonstrate their efficiency in image diffusion models. If not applicable, discuss why extending these approaches to the text-to-video diffusion model is not straightforward in the Related Work section, and also discuss the differences with existing baselines. - Week evaluation with DeepCache: It is unclear why the authors made a comparison with DeepCache under the same computational step. The authors should define what they mean by “computational step” and compare with DeepCache in terms of efficiency metrics as thoes in Table 1. Also, the metric of MAC and GFlops should be included to measure efficiency. - Evaluation protocol: The current evaluation using FVD and CLIP score is insufficient. It would be beneficial to include more comprehensive video quality evaluation metrics, such as thoes proposed in VBench [1]. - Additional experiments: Include more video diffusion backbones such ModelScope and VideoCrafter to extensively verify the effectiveness of proposed method under various video diffusion models. [1] Huang et al., VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024

Questions

What is the "computational step" in Table 2?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

See the Weaknesses Section.

Summary

The paper introduces a novel, training-free framework to optimize video diffusion models. This framework, consisting of Feature Slicer, Operator Grouping, and Step Rehash, significantly reduces peak memory usage and computational overhead while maintaining video quality. Extensive experiments demonstrate that the approach can cut memory usage by up to 70% and improve inference speed by 1.6 times compared to baseline methods. The framework is compatible with existing models like AnimateDiff and SVD, enabling high-quality video generation on consumer-grade GPUs. The research paves the way for more efficient video diffusion models, making advanced video generation accessible on standard hardware.

Strengths

- The introduction of a training-free framework that optimizes video diffusion models seems a novel approach. - The method is compatible with existing video diffusion models like AnimateDiff and SVD, ensuring broad applicability. - Comprehensive experiments and detailed analysis demonstrate the framework's effectiveness and robustness.

Weaknesses

- Although the paper claims that video quality is maintained, the extent of quality degradation, if any, is not fully quantified. - Although some ablation studies are provided, a more detailed breakdown of the contributions of each component (Feature Slicer, Operator Grouping, and Step Rehash) would strengthen the understanding of their individual and combined impacts. - The evaluation relies heavily on FVD and CLIP-Scores, which, while useful, may not capture all dimensions of video quality and user satisfaction. Including additional metrics or user studies could provide a more holistic assessment of the generated video quality.

Questions

- Are there any specific scenarios where the overhead introduced by slicing and grouping operations could outweigh the benefits? - Have you considered conducting user studies to provide a more comprehensive assessment of the generated video quality?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have adequately addressed the limitations.