COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing

Summary

The paper introduces COrrespondence-guided Video Editing (COVE), a method to improve video editing with pretrained text-to-image (T2I) diffusion models. It addresses the challenge of maintaining temporal consistency by using diffusion feature correspondence. COVE identifies and samples highly corresponding tokens across frames, applying self-attention to them during editing. It also reduces GPU memory usage with a temporal-dimensional token merging strategy. COVE integrates seamlessly into existing T2I models without extra training and shows superior performance in various scenarios.

Strengths

1. The method is reasonable and easy to follow. 2. The presentation is clear and easy to understand. 3. The Sliding-window Strategy seems useful in video editing methods.

Weaknesses

1. Experimental performance improvement seems limited. Most of the editing results present a similar type involving style/color change, which is easy to implement in the previous methods from my experience. According to Fig. 6, the performance compared to earlier methods does not show high superiority, which is not convincing to support "...COVE achieves the start-of-the-art performance...". 2. I noticed that [1] also uses DIFT features to guide video editing, especially object replacement. Could you please compare COVE with it? For example, replace the "jeep" with a "sports car" or "bus" in Fig. 11. I would like to see the performance when there is a large motion/shape change. [1] VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence

Questions

How do you make sure that $k=3$ is suitable for all videos? As suggested in Fig. 2, the subject may suffer from severe morph across video frames when there is a significant motion change.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

See weakness above.

Summary

In this paper, the authors tackle the problem of video editing using Text-to-Image diffusion models. To achieve this, the authors make use of strong diffusion model’s feature correspondence abilities. The authors propose a sliding-window based strategy to track features of source video based on correspondences. Here, the goal is to overcome the computational complexity involved with finding similarity across all patches of the video. This is achieved by computing similarity scores within a sliding window. After this, a set of corresponding tokens are collected and merged using an existing work (Token Merging paper, ICLR 2023). Experimental analysis shows that the proposed method achieves state-of-the-art performance across many tasks.

Strengths

S1. Model does not require any training or even inference time optimization. The method computes correspondences in an efficient manner and track it effectively. S2. The ideas proposed in the paper are simple and well presented overall. S3. Results look impressive - the method is able to achieve state-of-the-art in comparison to the works in the area of video editing with T2I models.

Weaknesses

W1. The paper clearly states that the base model is Stable diffusion on L208 and the model does not require any training including inflation (which I think is a major strength). Stable diffusion model has only self-attention and cross-attention layers. It is not clear how “temporal layers” come into picture on L200. Further, it is not clear what is the 3D-Unet in context of Fig. 3. W2. The proposed method tends to alter the regions of the image that do not correspond to the text prompt. In Fig. 1, the results shows that the proposed alters the background significantly. W3. Results show that the proposed method is unable to alter the shape of the object, since the correspondence merging limits the shape changing ability drastically. W4. The core novelty of exploiting the sliding window to use a lower field of comparison (as. In Eq. 10) is somewhat straightforward and limited given the obvious computational constraints. However, in conjunction with other ideas such as Token Merging [ref 1] and correspondences [ref 2], the work is novel. W5. There have been works in the video editing space which exploit correspondences for video editing [ref 3,4]. It will be helpful to differentiate the proposed method with them. However, I would like to acknowledge that these works could have been parallel to this submission. References [ref 1] Token Merging: Your ViT But Faster, ICLR 2023 [ref 2] Emergent Correspondence from Image Diffusion, NeurIPS 2023 [ref 3] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR [ref 4] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models, CVPRw

Questions

- How does applying \tilde{Corr} help during inversion?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper addresses the limitations of the work in Appendix E.

Summary

This paper focused on improving the temporal consistency of video editing. They propose to leverage the inherent diffusion feature correspondence with a sliding-window based strategy. With this design, the tokens in noisy latents can be sampled based on the “one-to-many” correspondence. The experiments demonstrate superior performance than other methods.

Strengths

+ The paper proposed a new diffusion feature correspondences guided video editing method. In addition, they introduce the token merging and sliding window for higher efficiency. + The method demonstrates superior performance than previous methods with extensive ablations to evaluate the effectiveness. + The paper is well-written with clear motivations.

Weaknesses

- The proposed method highly relies on the correspondences of diffusion features. However, such correspondences may be difficult to obtain for videos with large content motions. The reviewer suggests a more detailed discussion about how to ensure the accuracy of the correspondences, as well as what are the potential limitations of inaccurate correspondences. In addition, it would be better to quantitatively evaluate the correctness of the correspondences of diffusion features. - The paper mainly discussed and compared their method with optical flow-guided video editing methods. However, such correspondence-based idea is also related to deformation field based methods such as CoDeF [1] or neural atlas based methods [2], where they can ensure the accuracy of pixel / point level accuracy by evaluating the video reconstruction accuracy. The authors are suggested to compare or discuss their method with such approaches. - The related works seem to be a bit short and less extensive. There have been many video editing works that target at improving temporal / subject consistency, efficiency, long video editing, etc. The reviewer suggests to include a more extensive related works and discuss the differences of the proposed work with them. [1] Ouyang, Hao, et al. "Codef: Content deformation fields for temporally consistent video processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Chai, Wenhao, et al. "Stablevideo: Text-driven consistency-aware diffusion video editing." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Questions

o The reviewer appreciates the qualitative examples of correspondences of diffusion features in appendix. Is there any quantitative metric to evaluate the correspondences of diffusion features? o How to define the temporal length for token merging? Does it remain the same for all experiment videos?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper proposes using the correspondence features already exist in diffusion models to find matched tokens between different frames in a video for consistent video editing. The motivation is that video editing models using optical flow to find matched features can exhibit one-to-many matching issue which limits the quality of video editing consistency when there are large motions. To overcome expensive compute, the paper proposes using "sliding window" (w.r.t frames) by only finding correspondences in subsequent frames. Reasonable qualitative and quantitative results are shown in the experiments section.

Strengths

- The proposed method intuitively make sense - The flow of the paper is relatively easy to follow - The results shown are reasonable

Weaknesses

- Optical flow is also essentially just finding matched patches/features between different frames. The proposed way of finding one-to-many correspondences between frames can also be applied uusing optical flow (with some small modifications of existing optical flow algorithms). I do not see any evidence from the paper that using the features in a diffusion model is better than using the features extracted by an optical flow algorithm (explicitly or inexplicitly depending on different optical flow methods) - the above should be an important ablation study that is currently missing from the paper - the sliding window algorithm, though simple and achieving compute reduction, can also lead to errors accumulating across frames when editing long videos - the novelty is a bit limited. My argument is the following: 1. Features inexplicitly extracted in Stable Diffusion can be used for feature correspondence is well-known to the community, e.g. in Tang et al. NeurIPS 2023 2. Finding correspondences for video editing is widely studied and shown to be effective, including but not limited to the several references this paper already cited 3. The main contribution then seems to be proposing using SD's features to find correspondences instead of using optical flow. However, the one-to-many problem and many-to-one problem is essentially a thresholding problem when finding correspondences and the receptive field size used to find matched patches, and I cannot see why the features extracted by SD is superior than the features extracted by a state-of-the-art optical flow model. This ablation study is missing

Questions

See weakness

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

No