Towards Consistent Video Editing with Text-to-Image Diffusion Models

Summary

Previous works on one-shot video editing might produce results of unsatisfied consistency with text prompt as well as temporal sequence. This paper presents that such inconsistency is caused by covariate shift in the feature space. Therefore, they propose Shift-restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM) to improve the performance.

Strengths

1 The theoretical analysis of covariate shift is comprehensive and clear. 2 The proposed STAM can well address the style change problem while FFAM can improve the temporal consistency.

Weaknesses

1 The motivation for STAM is that the inconsistency between text prompt and video lies in the covariate shift between the pretrained T2I model and the fine-tuned T2V model. This paper provides a theoretical analysis of this. However, as the output layer of temporal attn is initialized as zeros and fine-tuned by a small learning rate, the influence of covariate shift may not be much significant. It is better to analyze the covariance quantitatively. 2 To address the covariance shift, in addition to the IC layer, spectral normalization is also proposed. But this paper does not provide an explanation and analysis of spectral normalization. 3 **Frame Consistency**: Fitting to the one-shot video improves the video consistency while STAM aligns the feature to the T2I distribution. Therefore, the two parts are contradictory, and the frame consistency is degraded by this. This is a severe limitation of this paper. In Table 1 and Figure 3, the temporal consistency of this paper is much worse than Tune-A-Video and Video-P2P. 4 The applications are limited. The proposed method mainly focuses on the style transfer problem. Other editing applications, such as background change, are not considered and compared with previous works. 5 Despite providing ablation videos in the supplementary material, the ablation studies in the main paper are not comprehensive. More video frames should be provided to show the influence on frame consistency.

Questions

My main concern is the frame consistency problem. The quantitative analysis of covariate shift and the explanation of spectral normalization are also important.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I think the frame consistency problem is a limitation that is ignored by the authors.

Summary

This paper carefully analyzes the problems of current one-shot video editing and proposes two techniques: Shift restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM), to address these issues. Several quantitative and qualitative experiments are conducted to verify the effectiveness of the proposed method.

Strengths

1. The authors meticulously analyze the cause of semantic disparity for video editing and attribute this to the covariate shift of generative feature space brought by newly added temporal attention. 2. To address this problem, the authors propose STAM to alleviate the covariate shift and further improve the temporal consistency with another technique FFAM. 3. The manuscript is well-organized and easy to follow.

Weaknesses

1. Is there any quantitative ablation comparison for the proposed techniques? It would be better to quantitatively analyze these components besides qualitative visual comparison. 2. From the visual ablation study, it seems that the normalized liner mapping contributes to the visual quality a lot. Is there any ablation that only applies normalized linear mapping?

Questions

see weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitation of the proposed method

Summary

This paper claims that existing models suffer from inconsistent video editing results due to covariate shift caused by additional modules for learning temporal information. It proposes a novel model addressing this issue through two modules: Shift-restricted Temporal Attention (STAM) and Fine-coarse Frame Attention (FFAM). STAM replaces Layer Normalization with an Instance Centering layer and utilizes a normalized mapping attention layer to preserve temporal feature distribution and constrain variance shift. FFAM incorporates fine-coarse spatial information to further improve temporal consistency. Experimental results demonstrate the effectiveness of the proposed EI model.

Strengths

1. The paper in detail analysed the issue of covariate shift in temporal consistency, which is a good perspective to cut in, and the proposed method accordingly tackled this issue. 2. Figures 4 and 5 show ablation experiments that illustrate how each component contribute to the final performance.

Weaknesses

1. Figure 3, as well as not few examples in the supplementary video (especially the first half almost), seems to show that the other methods aren't mostly doing well on text information, while having similar/comparable temporal consistency. This seems not to be consistent with the main target and point of the proposed method. In Table 1, the main metric, frame consistency in terms of CLIP score, doesn't show the proposed method's advantange either. I would like to learn more discussions here. 2. The examples in the supplementary videos are better to be listed together with the same input and all other methods, instead of separately one by one. Current layout only has one comparing method per slide, which is not clear and hard to compare with all.

Questions

1. [Minor] The title might be a bit too high-level since this paper just points out and targets at one aspect of the cause in temporal consistency issue. Addressing the specific technical points might be a more proper and solid claim.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

1. There are many latest related work on this topic emerging recently. Maybe many of them are concurrent work. It is not mandatory, but very good to include more comparisons with some most recent work, in terms of methodology, results and other technical differences, especially for those that have released all code and pretrained models.

Summary

The paper provides a comprehensive theoretical analysis of the covariate shift problem in current one-shot fine-tuning methods. In response, the authors propose a novel Shift-restrict Temporal Attention Module, designed to enhance the editing capability. Additionally, they introduce a Fine-coarse Frame Attention Module, which aims to improve temporal consistency. The visual examples presented in the paper demonstrate promising performance.

Strengths

1. The paper offers a comprehensive analysis of the underlying causes of semantic disparity in fine-tuning-based video editing approaches. Through a rigorous examination under the Gaussian assumption, the authors effectively demonstrate the occurrence of covariate shift. Furthermore, the proposed solution put forth in the paper is both logical and well-founded. 2. The paper showcases temporal consistency in the edited results, which is a noteworthy accomplishment. 3. The paper demonstrates a commendable level of clarity and cohesiveness in its presentation, making it easily comprehensible and accessible to readers.

Weaknesses

1. The analysis of covariate shift in this paper relies fundamentally on the assumption of i.i.d. Gaussian distribution. However, given the complex nature of real-world scenarios, this assumption may be overly restrictive. It would be valuable to incorporate a more generalized analysis to account for a broader range of situations. 2. While it is acknowledged that commonly used CLIP-based metrics may not align perfectly with human perception, they remain the most objective evaluation metrics available. Table 1 reveals that the CLIP score for temporal consistency of the proposed method is inferior to that of previous works. Could you please provide further details on the user study, particularly in terms of ensuring sufficient objectivity and randomness? 3. The visual examples depicted in Figure 3 highlight certain deficiencies in the fidelity of the proposed method to the original videos. For instance, the 3rd and 4th frames of the 'Spider-Man' case exhibit noticeable motion and pose disparities compared to the original frames. Additionally, the 4th frame in the 'man talking' case shows an open mouth, deviating from the original video. 4. In addition to assessing temporal consistency and textual alignment, it is crucial to incorporate fidelity evaluation metrics such as PSNR/SSIM/LPIPS as reported in Fatezero and Video-p2p. It would be beneficial to include these metrics in the analysis to provide a more comprehensive assessment of the proposed method. 5. The ablation study only includes one example to demonstrate the effectiveness of the proposed designs, which may not be sufficient. It is recommended to include a quantitative evaluation to strengthen the analysis and provide a more robust assessment of the proposed approach.

Questions

1. In L184, the authors state that randomly initializing the Temporal Attention (TA) module could potentially compromise its effectiveness in terms of editing capability. Considering this, would it be beneficial to explore the possibility of initializing the TA module from well-trained Spatial Attention (SA) modules, as shown useful in other works? 2. Upon reviewing the video comparisons provided in the supplementary materials, it becomes evident that several results from Video-p2p differ significantly from the demonstrations shown on their official website. For instance, the 'Spider-Man' riding case and the 'car driving' case display instances where the inversion step does not appear to have succeeded. I recommend investigating the code and corresponding results to address this discrepancy. 3. A minor suggestion regarding Figure 2: it may be preferable to present the 'Inflation' operation without the surrounding box. Including the box might lead to the misconception that this is a parametric module, which could be misleading for readers. 4. Only style manipulation cases are provided. Could the proposed method successfully address the pose/shape modification and object addition/remocal situations?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed the limitations.

Summary

This paper proposes $EI^2$ for the consistent text-driven video editing. To address the semantic disparity problem, it provides a detailed theoretical analysis and proposes Shift-restricted Temporal Attention Module (STAM) to tackle the covariate shift problem introduced by the temporal consistency module. To improve the temporal consistency, it provides a novel Fine-coarse Frame Attention Module (FFAM) to fully exploit the global temporal information while reduce the computational cost. Extensive comparisons with several available video editing methods shows its superiority and the ablation study also illustrates the effectiveness of the STAM and FFAM.

Strengths

- The authors provides a theoretical analysis about the semantic disparity problem in video editing, which makes the paper more convincing. - Based on the detailed analysis, the authors propose a novel Shift-restricted Temporal Attention Module to tackle the covariate shift, which can imrpove the semantic consistency in the edited video. - The design of the Fine-coarse Frame Attention Module makes sense, which can balance the performance and the computational cost. - The visual comparison shows its superiority over existing publicly available methods while the ablation study further illustrates the effectiveness of the proposed method. - The paper is well-presented.

Weaknesses

- Lack of quantitative results of the ablation study. Though the authors provide the qualitative comparsion result for the abltion study to validate the effectiveness of the STAM and FFAM, it will make the paper more convincing if they provide the some quantitative results for the ablation study. - Lack of some technical details. For example, when conducting the temporal attention, how many frames does the model take into consideration?

Questions

- The duration of the generated video is quite short (about 1s-2s). How does the proposed methods perform when generating long video? (upto 5s-10s) - Will the trianing and inference code be released?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have discussed the limitations in the main paper. More potential limirations please refer to Questions.

Final decision

Thanks for the authors' response. All my concerns are addressed.

About spectral normalization

Dear reviewer ipyC Thanks for your prompt response. We apologize for any confusion caused. To accommodate the word limit, we intend to utilize the reference to offer the implementation of spectral normalization (SN) within the context of STAM. Here we provide a comprehensive elucidation: - Formulation: SN offers a simple and straightforward way to normalize the matrix weight, denoted as $\bar{W} =W / \sigma_{\text{max}}(W)$, where $\sigma_{\text{max}}(W)$ is the max singular value, or called the spectral norm of $W$. After applying SN, we attain $\sigma_{\text{max}}(\bar{W})=1$, and $||\bar{W}||^2=1$ under the spectral norm. Drawing from Equation 7, where $||\Sigma_{rv(z)}' - \Sigma_{rv(z)}|| \leq || W_{L}||^2 || W_{v} ||^2 ||\Sigma_{rv(\hat{z})}||$, we primarily exploit SN to redefine linear operator within temporal attention. This will eliminate the influence of $W_{L}$ and $W_{v}$ on variance and make $||\Sigma_{rv(z)}' - \Sigma_{rv(z)}|| \leq ||\Sigma_{rv(\hat{z})}||$ in theory. - Implementation: Recognizing that computing exact singular values for high-dimensional matrices is often time-costing, we follow established works [1] by employing the Power iteration algorithm to fast approximate the max singular value. Consequently, the integration of spectral normalization does not result in additional time overhead. To avoid potential complications arising from small singular values during normalization, we practically apply spectral normalization solely to matrices with a spectral norm exceeding 1. - Effects: In most experiments for TAV, a distinct trend can be observed. Specifically, due to zero initialization, $||W_{L}||$ in temporal attention tends to concentrates within $(0,2)$, while $||W_{v}||$ tends to concentrates in $(3,6)$. Given the cumulative effect of squaring and layer-by-layer accumulation, the pivotal role of SN becomes evident in controlling these terms, validated by the quantitative analysis in A1. Moreover, the quantitative ablation study provided in table 1 of the PDF file indicates that, directly leveraging Normalized Linear Mapping (NLM) can obviously improve the textual alignment score (29.02) compared with baseline model (+FFAM 28.50). The qualitative results in Figure 4 also validates this point. Thus, SN does play an important role in alleviating the covariance shift. We are pleased to offer further explanation on any remaining questions you might still have. [1] Miyato, Takeru, Toshiki Kataoka, Masanori Koyama and Yuichi Yoshida. “Spectral Normalization for Generative Adversarial Networks.” ICLR 2018.

Post-rebuttal Comment

Thanks for the response. All of my concerns have been addressed. I will keep my origianl rating.

Help check if questions are well addressed.

Dear Reviewer ipyC Thanks for your efforts and suggestions again! Could we kindly know if the responses have addressed your concerns and if further explanations or clarifications are needed? Your time and efforts in evaluating our work are appreciated greatly.