VideoComposer: Compositional Video Synthesis with Motion Controllability

Summary

This work aims to allows users to flexibly compose a video with textual conditions, spatial conditions, and temporal conditions. It introduces a novel framework namely VideoComposer based on the paradigm of compositional generation. To be specific, it introduces the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. Moreover, it develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experiments demonstrate that VideoComposer control the spatial and temporal patterns simultaneously within a synthesized video in various forms.

Strengths

1. It introduces motion vector as a more flexible user-guided signal. 2. It proposes Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs. 3. Extensive experiments show the effectiveness and superiority of VideoComposer.

Weaknesses

1. What is the difference between the roles of the ``Style`` of CLIP and ``Single Image`` of STC-encoder? They both seem to provide content to videos. 2. VideoComposer only obtain comparable performance with prior video generative models. Is it more efficient than previous methods? The authors could give their comparisons in training cost and inference time. 3. Lack of extensive visualization comparisons with existing video generative models. The authors are encouraged to provide extensive qualitative comparisons in video generation task.

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes.

Summary

This work proposes a new method called VideoComposer for conditional video generation, especially for video-to-video translation. VideoComposer is constructed upon the Video Latent Diffusion Model and introduces an STC-encoder to integrate multiple spatial and temporal conditions such as RGB images, sketches, motion vector sequences, etc. The architecture design involves simple 2D convolutions and temporal transformer layers. The conditional features are fed into the U-Net input together with noise. The demonstrated results have good temporal consistency.

Strengths

- This is one of the pioneering works in controllable video synthesis. The temporal consistency of the video results is impressive, considering that its conditioning modeling enables several editing abilities such as image-to-video translation and motion/depth/sketch-driven local/global video editing. - The jointly training strategy is good for flexible inference within one model, e.g., video inpainting, without second training. - The paper organization and illustrations are easy to follow.

Weaknesses

- The authors could have tried other design choices for integrating Condition Fusion as input into the U-Net, such as integration through cross-attention. - In line 215, it is claimed that “we observe that the inclusion of mask and style guidance can facilitate structure and style control.” However, the corresponding evidence should be presented for the style representation extracted by clip image encoder and concatenated with text embedding. - It seems that a single STC-encoder is used for all different conditions via random dropout. It would be interesting to see if different STC-encoder weights for different conditions are better. - The examples in Figure 6 with reference image look like failure cases. Besides, the tiger texture and box shape are changed in Figure 8. It would be helpful to see more discussion and analysis on this part. - The ablation study of STC-encoder is not presented in a fair way. The main benefit of using STC-encoder comes from the video information condition instead of the network design. - The important comparisons and discussions with other methods are not sufficient, such as VideoP2P and vid2vid-zero mentioned in the related works.

Questions

The comparisons with other methods are not sufficient and the ablation study is not well presented.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The societal impact has been discussed.

Summary

VideoComposer is a tool designed to enhance video synthesis by incorporating textual, spatial, and temporal conditions. It uses motion vectors from compressed videos to guide temporal dynamics and employs a Spatio-Temporal Condition encoder to effectively integrate spatial and temporal relations of inputs. This improves inter-frame consistency and allows for greater control over the synthesized video's spatial and temporal patterns.

Strengths

The VideoComposer offers better control over video synthesis, temporal guidance using motion vectors, improved inter-frame consistency with its Spatio-Temporal Condition encoder, versatility in accepting various forms of inputs, and high customizability, resulting in more precise and desired synthesized videos.

Weaknesses

An ablation study could be conducted on VideoComposer, where each component is removed in turn to evaluate its impact on overall performance. This would help evaluate the value of training under multiple conditions versus a single condition. Additionally, comparing VideoComposer to a simpler method like Text2Video-Zero [a] with ControlNet [b] would demonstrate whether the increased complexity of VideoComposer yields significantly better results, hence justifying its sophistication. [a] Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, L Khachatryan et al. [b] Adding Conditional Control to Text-to-Image Diffusion Models, L. Zhang et al.

Questions

How are image-text pairs utilized in the training process of the model?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

What are the limitations?

Summary

The paper presents a method for compositional video synthesis. It introduces motion vectors from compressed videos as a control signal for temporal dynamics. The motion vector can be combined by other conditions such as sketch, and depth map. Both qualitative and quantitative results show that the proposed method can control the spatial-temporal patterns.

Strengths

+ The motion controlled generation result (fig 8) using hand-crafted strokes is interesting. + Table A1 shows effectiveness of the proposed method quantitatively compared to previous methods. + The paper is well written and easy to follow.

Weaknesses

- There are a few GAN-based video synthesis approaches that are worth discussing in the related work. For example, MoCoGAN [1] approaches the problem by decomposing motion and content. - The two-stage training strategy needs more clarification. What is "compositional training" particularly in the second stage? How does it differentiate from the "text-to-video" generation in the first stage? - In line 164-165, the authors "repeat the spatial conditions of a single image and single sketch along the temporal dimension". If the input condition is simply repeated, what's the point of applying a temporal Transformer? It will be equivalent to applying the spatial operation only and repeat at the latent space but with higher computation cost, no? (for motion vector, I totally agree that a spatial-temporal modeling would be necessary.) - Motion vectors can be less meaningful in the background due to lack of high-level semantics. It can also be clearly seen from the top row in Fig 4. I wonder if the authors treat the motion vector field equally for all locations. It seems that the generated results with motion conditions has more blurry background. - From Figure 2 and Figure 1(d), my impression is that the conditions (say motion and depth) can be combined together. However, in ablation studies (table 2), only one condition is added at a time. Another ablation that studies all combinations of these conditions will be favored. [1] Tulyakov, Sergey, et al. "Mocogan: Decomposing motion and content for video generation." CVPR 2018.

Questions

1. In the video translation demo (2:11-2:16), the right example's output does not have a consistent color (white before jumping and brown afterwards). Is there any particular reason why the color consistency fails to hold in such a case? 2. I'd suggest moving some quantitative results (Table A1) to the main text.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have addressed the limitations in the supplementary materials.

Thanks for reponse

Thanks for the author's elaborate response, and all my concerns have been well addressed.