SF-V: Single Forward Video Generation Model

Summary

This paper proposes a method to accelerate video generation model inference speed by distilling the multi-step reasoning of Singular Value Decomposition (SVD) into a single-step generation using adversarial networks. This approach achieves comparable results to multi-step SVD generation while significantly improving inference efficiency. Experimental results validate the effectiveness of the authors' method.

Strengths

1. The algorithm proposed by the authors significantly improves video generation inference speed without compromising the quality of the generated results. 2. The authors propose using two parallel heads, a spatial head and a temporal head, to implement the discriminator. This resolves the issue of a single head potentially leading to the generation of static videos.

Weaknesses

1. The authors trained the entire network using 1 million internal videos. For those without access to this dataset, reproducing this work could be challenging. Will the authors make these data and source code publicly available? 2. How many training data points are needed to ensure that the distilled model has strong generalization capabilities? The training process for SVD likely requires far more than 1 million data. Is using 1 million data for distillation training sufficient? 3. The author mentioned using the encoder of a Unet for initializing the discriminator. What is the insight behind this choice? Why did they consider this initialization method? 4. In Adversarial Diffusion Distillation, they use distillation loss to transfer knowledge from the teacher model to the student model. Why did the authors not use this approach here? Is there any specific reason?

Questions

see Weaknesses

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

It's unclear how well the method generalizes currently, as the authors only conducted quantitative analysis on the UCF101 dataset. This uncertainty arises because the authors trained the model using only 1 million data points.

Summary

The paper tackles the task of distilling diffusion-based text-to-video models into single-step models, achieving much higher sampling speeds. To this end, they build a framework to fine-tune a pretrained video diffusion model. This fine-tuning is done in an adversarial setting in the latent space, whereby the generator and discriminator are designed to achieve higher image quality and video consistency. In particular, the generator is initialized from a pretrained video model, and the discriminator is partially initialized with the layers of a pretrained encoder, where additional spatial-temporal layers are added and trained. The model achieves SOTA SVD quality in the setting of 1-step video diffusion and an additional speedup compared to existing art.

Strengths

The paper is very well written and easy to read. In particular, the introduction and related work well put the work in context and the contribution. The related work section is sufficiently extensive and provides the right context to identify the existing gap the paper is trying to solve. The results and visualizations are clear, well-explained, and detailed. The paper is also well validated, showing excellent results compared to the state-of-the-art in single-step video generation, achieving superiority in terms of quality as well as sampling time. By fine-tuning an existing video model, the authors can also achieve good training time and use any knowledge captured by the pretrained model. In that sense, the proposed approach makes sense and is well-suited for the task.

Weaknesses

My main concerns are concerning the novelty and significance of the proposed approach in the following sense: 1. One component of the method is a re-design of the generator and discriminator. First, this in itself seems to be of limited novelty. The proposed components are standard and well-established in the community. Finetnuning the discriminator by adding additional layers is not new, and similarly, for the generator, copying the pretrained encoder seems sensible but does not offer a new contribution. Second, one could take the existing single-step adversarial text-to-image approaches and replace the generator and discriminator with the architecture proposed in this paper. The authors identify such papers well (e.g. [24, 25, 26, 27, 28]) but do not compare them to such a baseline. Hence, assessing whether the proposed architectural changes are meaningful is difficult. 2. The proposed training regime (adversarial training and reconstruction losses) is another component. On the flip side to 1, is this training regime important? In that sense, one could take existing single or multi-step video diffusion baselines (e.g., [14, 21]) and change the training regime to that proposed in this paper. Further, assuming that this training regime improves in the video setting, one could assume that it is more general and applicable also in the single-step text-to-image regime. Further, the proposed components both seem to be marginally novel. Adversarial training for single-step distillation and the temporal components are also not new. Is there something non-trivial in the combination of both components?

Questions

I believe the paper provides a great engineering effort, significantly improving sampling speed and quality compared to the state-of-the-art. However, I am concerned about the novelty of the proposed approach, as indicated in the weaknesses section.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Limitations are briefly addressed but not sufficiently analyzed. It would be great to see examples of errors introduced by the approach or some further analysis.

Summary

The authors propose a method to generate similar-quality as the original video diffusion model Stable Video Diffusion (SVD) in a single feedforward pass. To this end, they take the pre-trained SVD model and fine-tune it with a reconstruction and adversarial loss. The discriminator uses a frozen copy of the SVD encoder along with trained spatial & temporal readout heads to encourage both spatial and temporal consistency. Further, the authors add noise to the inputs of the discriminator to stability training. The results are of slight better quality than 16 steps of SVD, and slightly worse than 25 steps of SVD.

Strengths

- The paper is very clear and easy to read - The presented approach is simple - It works! Quality is somewhat comparable to several diffusion steps with the original model

Weaknesses

- A large chunk of the methods section repeats how the original diffusion model is trained -- since this is already known information and not directly relevant to the presented method, it could be shortened or moved to the appendix - The proposed method is fairly simple and incremental, though the results are better than existing methods

Questions

- L257: what's the relative compute / time that goes into the encoder / decoder compared to the actual latent model?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors briefly discuss the fact that the VAE decoder & encoder for image conditioning. It should be noted that training GANs can often come with the additional headache of instabilities.

Summary

The paper proposes an idea of training a distillation approach using GAN based technique. The advantage which is suggested by the authors is that such distillation approach can reduce the computational cost associated with the sampling new samples during the inference time. Instead of taking multiple sampling steps only single step is required for inference.

Strengths

The idea is sound and well founded. Utilizing GAN based distillation approach to reduce the computational overhead related to the sampling process in diffusion models. This is especially important for videos since a minute long video consists of almost 1.5k frames.

Weaknesses

- Videos do not look like they're real world videos. How does this work with the real world videos. - The motion is very laminar in the videos. How is the performance when trained on videos with non laminar motion.

Questions

- GANs suffer from mode collapse problem how was it ensured that is not the case here with this approach? - GANs suffer from the problem of instability in training how was it handled - Why weren't standard video datasets like UCF101 used in evaluation?

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

GAN based models suffers from training instability because of the presence of adversarial loss.