Video Diffusion Models are Training-free Motion Interpreter and Controller

Summary

This paper introduces a new Motion Feature (MOFT) that can effectively capture motion information in video diffusion models. The authors reveal that robust motion-aware features already exist in video diffusion models, allowing to encode comprehensive motion information with clear interpretability. They present MOFT, which can be extracted without the need for training and is generalizable across diverse architectures.

Strengths

- Training-free strategy effectively extracts motion information encoded in the features of the video diffusion model, demonstrating its ability to capture and leverage the inherent motion representations learned by the model. - The method presents a clean and straightforward solution for extracting motion encoding from video diffusion models, making it a ready and practical technique for various applications involving motion analysis or synthesis.

Weaknesses

- The paper lacks clarity on the training process. While it claims to be training-free, it defines loss functions for other tasks (Equations 3 and 4). It would be helpful to clarify which stages are trained and which are not. - The PCA analysis is based on a small number of videos (only 2 videos in Figure 2), which limits the generalizability of the results. - While the motion in the qualitative videos looks good, the differences compared to other alterations appear subtle and hard to recognize. Other methods show too poor results, were they tuned correctly? - The paper should report the runtime and resolution for better understanding of the method's computational requirements and output quality. - The idea is heavily inspired by DIFT and utilized for video applications, then novelty seems limited.

Questions

Clarification needed: - Figure 1: a. It is unclear whether the motion feature in 1(a) is extracted semantically or spatially. Clarification is needed on how the similarity with other videos in 1(b) is calculated. Additionally, an explanation of what the higher score represents and why motion features from different videos could influence each other would be helpful. b. In 1(c), the motion direction seems to be manually defined. If so, why does the paper state that MOFT serves as guidance for controlling motion direction? If MOFT controls the motion, what is the source video for that motion? - Figure 6: Why the comparison is presented in the form of a point for DIFT and a segment for MOFT.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are addressed adequately.

Summary

The paper introduces a training-free framework for understanding and controlling motion in video diffusion models. The key innovation is the MOtion FeaTure (MOFT), which is derived by removing content correlation and filtering motion channels from pre-trained diffusion model features. MOFT provides a training-free way to encode and manipulate motion information, offering high interpretability and generalizability across various architectures. The framework demonstrates competitive performance in generating natural and faithful motion, with applications in video motion control and point-drag manipulation.

Strengths

1. Training-free Approach: The framework does not require additional training, leveraging pre-trained diffusion models to control motion, significantly reducing resource requirements. 2. Interpretability: MOFT offers a clear and interpretable way to understand and manipulate motion information in video diffusion models. 3. Generalizability: The method is applicable across various video generation models, demonstrating versatility and robustness.

Weaknesses

Scalability to Longer Videos: The proposed method's scalability to longer videos or higher resolutions is not adequately explored. Complexity of MOFT Extraction: Removing content correlation and filtering motion channels may be complex and require fine-tuning for optimal results. Experiments: The text prompts used for quantitative experiments and user studies are unclear, and 56 case studies are insufficient to validate effectiveness.

Questions

Providing a detailed description or pseudocode for MOFT extraction would aid in understanding its practical implementation and reproducibility. Have you tested MOFT's scalability for generating longer videos or higher-resolution outputs? What challenges, if any, did you encounter, and how did you address them? Additionally, how does this method handle more complex object control, such as multi-object scenarios with different categories and sizes?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The author have discussed limitations in Supp.

Summary

This paper investigates the relationship between the features of video diffusion models and the motion in the generated videos. By extracting motion features and using them as guidance, training-free motion control can be achieved.

Strengths

1. The technical aspects of this paper are clear and it is easy to read. 2. The proposed method can achieve training-free motion control for video generation. 3. The framework can be applied to different forms of control signals.

Weaknesses

1. From Fig. 6, it is hard to draw the conclusion that "MOFT can provide more valid information than DIFT at the early design stages". 2. The analysis experiments in the method section only focused on very simple motions, such as pan up, down, left, and right, without discussing more complex and realistic motions. 3. The generated motions presented in the experiment section are also mostly very simple. 4. The experiments lack comparisons with existing methods. Comparisons with other methods were only made in the point drag mode.

Questions

Please refer to the waknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are discussed while the potential negative societal impact is not discussed. But I don't think this discussion is necessary.

Summary

This paper presents a training-free method for motion control in video diffusion models and explores the interpretability of features within these models. The authors demonstrate through experiments that principal components of the features, extracted using PCA, contain motion information. They propose a pipeline that eliminates content correlation information from the features, filters motion channels, and optimizes the initial latent input in the diffusion model's UNet.

Strengths

- The paper effectively demonstrates the potential of internal features in video diffusion models to capture motion information. - The use of PCA to eliminate irrelevant information is well-justified. - The proposed training-free method can be applied to various manipulation scenarios, including reference-based and drag-based control.

Weaknesses

- In Section 3, the authors discuss the challenge of extracting motion information from diffusion features due to their encapsulation of other data types, such as semantic and structural correlations. The paper lacks a detailed explanation of "content correlation information" (section 3). It is unclear whether this term encompasses semantic, structural, appearance, background, lighting, or other information. - Figure 6 contrasts the video motion control capabilities of DIFT and MOFT. It is recommended that identical sample images be used and their similarity heatmaps be compared, thereby better visualizing the capability gap between DIFT and MOFT. - The optimization process of the latent in Section 4.1 could be more clearly explained, particularly the settings from references [31; 41]. - A more descriptive caption for the Motion Control Pipeline (Figure 5) and a clearer title for the caption of Figure 6 are advised for better comprehension.

Questions

Q1. After eliminating content correlation information, does the feature retain any information other than motion? If so, could this residual information affect video generation? Q2. Figure 6 illustrates how the ability to track motion varies with generation steps. Is it possible to manipulate features from earlier generation steps to achieve better results?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Not included. While the research demonstrates the potential of training-free methods to encode motion information, it remains unclear whether the proposed method completely eliminates motion-correlated information or retains any motion-irrelevant information.

Thanks for your rebuttal.

The authors have addressed most of my concerns. I decide to keep my initial positive rating.