Summary
The paper proposes Live2Diff, a video diffusion model that utilizes uni-directional temporal attention as well as additional warmup and depth information for streaming video translation. In specific, the uni-directional attention is conditioned on a sliding window of a few preceding frames, and the warmup contains attention from a small number of frames at the beginning of the video. To achieve real-time inference, Live2Diff uses three techniques: First, it replaces vanilla diffusion models with LCM-LoRA to reduce the number of denoising steps. Second, it uses a KV-cache during the implementation to avoid redundant computation. Third, it designs a pipelined denoising approach to accelerate model inference. In the experiments, the author(s) compare Live2Diff with three video translation methods, including StreamDiffusion, Rerender, and FreeNoise, and demonstrate that Live2Diff achieves higher CLIP scores and lower warp error while requiring lower latency.
Weaknesses
1. I have some doubts about the novelty in this paper. In specific, just like the authors mentioned, several works have adopted an autoregressive way for long video generation, e.g. ART-V, SEINE, Gen-L-Video, Reuse and Diffuse. The idea of incorporating a small number of frames in the beginning is also not new [1]. The author(s) claim their major contributions of proposing uni-directional attention with warmup. However, it seems naive to me that the author(s) simply replace bidirectional attentions with uni-directional attentions because of the problem setting, where the target frame shall not have seen the later frames and can only use the information from preceding frames. Regarding warmup, it can be seen in Table 2 that warmup does not provide significant improvements. (Row A, B, and C have similar values.) Instead, the improvement comes from using additional depth priors instead of model architecture.
2. The selected baselines are somehow outdated. Similar work, such as StreamV2V (Open source, Arxiv, May 2024), that also targets real-time streaming video should be included for comparison. State-of-the-art video translation methods, like CoDeF and FlowVid, which may have better temporal consistency, editing performance, but lower throughput, should also be compared. In specific, StreamV2V lies in exactly the same scope as the paper, and they compare with both CoDeF and FlowVid, both of which outperform the selected baselines, i.e. Rerender and FreeNoise, with much better temporal consistency.
3. The literature review is not complete enough. There are several missing methods for video processing. To my best knowledge, existing methods can mainly be classified into four categories: propagation-based, representation-based, canonical-based, and attention-based. These methods use different strategies to improve temporal consistency, like propagation across frames [2,3], video deformation [4,5], transferring video to canonical space [6,7], or using attention mechanisms. It is acceptable that the paper only focuses on attention-based methods, which is much closer to the scope, but there are still many missing approaches that use diffusion models for video generation/editing. Some of them are zero-shot methods that do not require model training, e.g. Controlvideo, Tokenflow, Control-A-Video, VideoControlnet, Text2video-zero. Some of them further enhance temporal consistency with cross-attention, e.g. Tune-A-Video, Text2Video-Zero, FateZero, Vid2Vid-Zero. Some works propose training for video editing, e.g. Imagen Video, Make-A-Video. In short, I suggest the author(s) provide more comprehensive literature review in Section 2.
4. The temporal consistency is bad in some examples demonstrated. (Comparison 1, 2, 3, 6) To my knowledge, this is because the backgrounds in these examples contain more 'high-frequency' details. In other words, they are more complicated, compared to other 'dancing human' videos in indoor scenes with simple background. Since the paper has incorporated the depth priors, is it possible to first differentiate the static (background) and dynamic objects (human subjects, for example) in the video, and then do a simple masking techniques to avoid temporal inconsistency in the static background?
[1] W Harvey et al., Flexible Diffusion Modeling of Long Videos, NeurIPS 2022.
[2] O Jamriška, Stylizing video by example. ACM TOG 2019.
[3] X Wang et al., Learning Correspondence From the Cycle-Consistency of Time, CVPR 2019.
[4] V Ye et al., Deformable Sprites for Unsupervised Video Decomposition, CVPR 2022.
[5] Y Kasten et al., Layered neural atlases for consistent video editing, ACM TOG 2021.
[6] H Ouyang et al., CoDeF: Content Deformation Fields for Temporally Consistent Video Processing, CVPR 2024
[7] T Chen et al., NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing, NeurIPS 2024.
Questions
1. Figure 3 and Figure 6 do not demonstrate strong initiatives in using depth priors. Since the baselines may not have depth priors, I am curious about the qualitative comparison between baselines, uni-directional attention with warmup, and uni-directional attention with warmup and depth.
2. Could you provide the video results of Figure 3? It seems that the full model with Figure 3(f), i.e. full model with depth, has worse temporal consistency than Figure 3(e), i.e. full model without depth, specifically at the regions of human clothes and face. I expect serious jittering artifacts in Figure 3(f).
3. Could you provide the comparison with more baselines, especially StreamV2V?
4. It seems to me that all the generated results are very similar to the input videos, and the prompt must align with the input videos. Is it possible for Live2Diff to do other editing tasks besides style transfer, for example, changing the color of human clothes?
5. See weakness.