Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning

Summary

This paper introduces DEMO, a text to video diffusion approach that aims at enhancing the movement in the video produced by text to video models. In additional to the classical diffusion loss, DEMO introduces: * __A text motion loss__: the CLIP text encoder used for the temporal cross attention is fine-tuned - the loss matches the text motion to the motion of the video as computed with the optical flow * __A regularization__: to avoid the CLIP text encoder used for the temporal cross attention to diverge too much from the CLIP encoder used for the spatial cross attention * __A video motion loss__: to encourage the frame latents output by model to follow the difference between frames in the actual video The model is trained on WebVid and compared to other models trained on WebVid on both video quality and motion quality.

Strengths

Technical contribution is strong: * The need for motion focused text encoder is well argued * The experiments show that this addition contributes into increasing movement Technical clarity is also good: * The different losses are presented in a clear and detailed way.

Weaknesses

The comparison with other approaches is not 100% fair in the sense that additional parameters are being trained on the text encoder side (a CLIP encoder). It might also be that additional parameters are being trained on the temporal cross attention side. This is actually unclear to me, because at the beginning of the method section (section 3) the author define the U-Net 3D this way: "the temporal transformer consists of temporal self-attention and feed-forward layers", while some papers do use cross-attention in the temporal self-attention as well.

Questions

The author define the U-Net 3D this way: "the temporal transformer consists of temporal self-attention and feed-forward layers", while some papers do use cross-attention in the temporal self-attention as well. Is that a mistake in the text, or to which specific papers are you referring to as the "vanilla" 3D U-Net ? Is the temporal attention similar to Make-A-Video and is operating on a single patch ? Or do you use full attention across all patches of all other frames ? That also has an impact on the fairness of comparisons / ablations of the importance of the losses that you introduce.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

n/a

Summary

This paper introduces novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Authors investigate sensitivity of CLIP text encoder to motion descriptions and propose to condition the model on content and motion separately. To obtain a motion encoder, authors propose fine-tune separate CLIP text encoder with losses that encourage its [eot] token to describe motion better. To this end, authors introduce novel text-motion and video-motion losses to encourage the motion conditioning module to generate and render motion dynamics. Experiment results show that proposed method enhances motion of the T2V model.

Strengths

1. Importance. The proposed method successfully tackles the an important problem of enhancing motion quality in videos generated by T2V models. 2. Results. Experiment results demonstrate the effectiveness of the proposed method. 3. Clarity. The text of the paper is well written and easy to follow.

Weaknesses

1. Lack of user study. Qualitative results are supported only by examples of several videos. Such results are not statistically representative. Please See Question 1.

Questions

1. I strongly recommend that the authors incorporate a user study in their research to obtain qualitative results. Additionally, conducting ablation studies with a larger sample size, consisting of a minimum of 50 distinct prompts, and involving at least 15 independent assessors, would greatly enhance the robustness and reliability of the findings. 2. On line 127, the authors mention calculating optical flow between attention maps of different frames. However, the methodology employed to perform this operation lacks clarity in the paper. It is essential for the authors to provide a more detailed explanation of the procedures used to compute optical flow between attention maps. Furthermore, considering that attention maps in diffusion models are often noisy, it is crucial to address the reliability and potential limitations associated with this computation.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations.

Summary

This paper proposes a method for enhancing motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components, called Decomposed Motion (DEMO). To address the issue of inadequate motion representation in text encoding, they decompose text encoding into content encoding and motion encoding processes, focusing on spatial information and temporal information respectively. To solve the problem of reliance on spatial-only text conditioning, they also decompose the text conditioning process into content and motion dimensions to integrate spatial and temporal conditions. Additionally, they incorporate text-motion and video-motion supervision to enhance the model’s understanding and generation of motion.

Strengths

1. The paper is well written overall and it is easy to follow along with the presented concepts and results. 2. The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding of model design choices. 3. The motivation for proposing the method is well demonstrated experimentally. 4. The proposed method is efficient in training, has a small number of parameters and a low inference burden, and does not require additional reference information.

Weaknesses

1. There is too little content in the related work section. 2. There is a lack of contrast experiments with other improved methods that rely on external references to exhibit performance differences. 3. The proposed method is generalized, so why not validate it on other BASE models? From qualitative and quantitative experiments, the performance and various indexes of ModelScope are poor, and while DEMO is useful, it is unclear if there is any improvement when adding DEMO to models with better performance. There is a lack of relevant experimental evidence. 4. The provided videos are less impressive. The separate files of other methods make it difficult for reviewers to identify the advantages of the proposed method, it would be better to make the comparison in a single file.

Questions

In line 196, “showing marked improvement in individual frame quality compared to the baseline,” can you explain why the quality of a single frame is increased? As I understand, DEMO is designed to increase the model's motion generation capability.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness.

Summary

This paper aims to improve the motion dynamics generation of the text-to-video generation models. It proposes a framework to decompose both text encoding and conditioning into content and motion components. For text encoding, a CLIP encoder is fine-tuned to encode the motion information in the text prompts better. For text conditioning, it introduces a motion conditioning module to incorporate motion information in the denoising process.

Strengths

1. The general idea is interesting, i.e. decompose the text encoding and conditioning into content and motion components to capture the motion prompts and generate motion dynamics better. 2. The pilot study is interesting, which investigates that the CLIP encoder tends to be less sensitive to the motion instructions in the text prompts. 3. This paper is technically clear and easy to follow.

Weaknesses

1. For the pilot study in Fig. 1, only one kind of sentence template is applied. Will the drawn conclusion be the same when the text prompts changes to other formats? This concern is not addressed in the paper. 2. In Fig.3, it is hard to tell which rows are generated by the base model and which rows are generated by the fine-tuned model. 3. From line 179 to line 183, it is hard to evaluate the correctness of the conclusion. Because it is quite hard to distinguish the two generated videos shown in Fig. 3. They look almost the same. 4. For qualitative evaluations. The improvements in generated motions in most shown cases are limited.

Questions

Please refer to the weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are discussed in the paper.

Thanks for the rebuttal.

The authors have addressed most of my concerns. I decide to keep my initial positive rating.

Answer to Rebuttal

Thanks a lot to the authors for answering my concerns. I will keep my rating the same.