Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models

Summary

This paper aims to address video inverse problems using image diffusion models. By viewing videos as sequences of warping transformations between frames, the authors propose a novel approach called warped diffusion from a continuous function space perspective. Specifically, warped diffusion includes a Gaussian process-based noise-warping technique and a post-hoc equivariance self-guidance method to enhance temporal consistency. Experiments on video super-resolution and inpainting validate the effectiveness of warped diffusion. Moreover, warped diffusion first demonstrates superior performance with latent diffusion models such as SDXL.

Strengths

1. The proposed method is well-motivated with well-constructed formulations. 2. The paper is well-written. 3. The visualization results are impressive.

Weaknesses

1. As illustrated in Table 1 and Figure 3, the proposed GP Noise Warping performs worse than "How I Warped Your Noise" in terms of warping error. The core performance improvement appears to come from the equivariance self-guidance. 2. As mentioned in Section G.3, the equivariance self-guidance is inefficient. Although the proposed GP Noise Warping is more efficient than "How I Warped Your Noise," the overall process can be more time-consuming.

Questions

1. "How I Warped Your Noise" operates in a zero-shot manner using pretrained video editing diffusion models without requiring additional training. Is it possible for warped diffusion to also function in a zero-shot manner? 2. I am curious about the effectiveness of warped diffusion with just the equivariance self-guidance alone, without the GP noise warping. 3. What would the performance be like if the equivariance self-guidance technique were integrated into "How I Warped Your Noise"? Would it surpass the current performance of warped diffusion?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations and societal impacts have already been discussed in the paper.

Summary

This paper addresses the challenge of temporally correlated inverse problems by employing an image diffusion model. The authors introduce a technique termed Warped Diffusion, which incorporates an equivariance self-guidance mechanism. This mechanism ensures that the generated frames maintain consistency when subjected to warping transformations. The paper demonstrates the application of this technique in the context of video inpainting and video super-resolution through a series of experiments.

Strengths

1. The paper is presented in a structured manner, with illustrations that aid in understanding. The content is coherent and follows a logical sequence. 2. The authors offer an analysis of noise warping and equivariance self-guidance, providing both intuitive explanations and theoretical underpinnings. 3. 3. The experiments are extensive to support the claim.

Weaknesses

1. The authors report the noise warping speed at Line 357. We recommend that the authors also include the complete inference time required to process an entire video, which would provide a more comprehensive understanding of the method's efficiency. 2. We suggest that the authors provide more video results to further demonstrate the capabilities and limitations of the proposed method in various scenarios. 3. It is observed that the inpainted regions in the video inpainting examples remain static, such as the cat in the video not exhibiting motion. All apparent motion seems to be attributed to camera movements, with no optical flow present within the inpainted regions. We would appreciate an explanation for this phenomenon and its implications on the video inpainting process.

Questions

See the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately discussed the limitations and negative societal impact in the paper.

Summary

The paper follows a previous work on noise-warping methods for video inverse problem. The authors proposed instead of giving temporally consistent noise maps to the DM, using a continuous function space representation for DM directly can serve more complex spatial transformations and results in better temporal consistency in the generated videos. They demonstrated the proposed method on video inpainting and super-resolution.

Strengths

The proposed method is novel in terms of using Gaussian processes to handle the noise function so that no mapping is required for the noise. The authors derived the equivariance for deformation and by satisfying this the gaussian process is used to guide the generation to be consistent under the warping transformation. To avoid additional training, a sampling mechanism was proposed to ensure this satisfaction at inference time.

Weaknesses

The key statement in the paper is the importance of the equivariance of the DM to achieve temporal consistancy. The authors argued that the previous work "How I Warped Your Noise" performed badly when the DM is not equivariant. However, there are no explicit examples or solid proof on this statement. The proposed method for noise warping is more like offering another perspective than showing the failing reasons of the other work. Furthermore, for super-resolution experiments, the authors did not provide comparison against "How I warped your noise". Even though they elaborated that the authors of "How I warped your noise" acknowledged to them that the work does not result in temporal consistency, I believe it is still better to include the comparison given how many super-resolution cases were presented in that work. If they don't provide the code, the authors may consider modify the one they have for the inpainting experiments and list out the comparison for more convincing results.

Questions

1. It is not clear to me why the function space diffusion models need to be equivariant with respect to the underlying spatial transformations. I can imagine the necessity, but in your derivation, I don't see the proof that you stated you provide. Can you provide explicit examples or solid proof on this statement? 2. Why in fig 3 how i warped your noise seems to be in the second place in terms of error, but in table 2, the method seems not to outperform any other baselines?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors did not address any limitations

Summary

This paper proposes a self-guidance equivariance approach using the Image Diffusion model for generating temporally consistent videos. In previous work HOW I WARPED YOUR NOISE[1], noise is warped across frames to ensure the noise maps are temporally consistent. However, temporally consistent noise maps do not guarantee temporally consistent output. In this work, the authors address this issue. The proposed self-guidance equivariance enforces that the generated frames remain consistent under the warping transformation. The proposed method can generate more consistent results. In their experiments, the authors conducted comparisons on video inpainting and superresolution. It is worth noting that the proposed method outputs consistent content in the inpainting area across frames.

Strengths

- Problem Setup - The problem seems reasonable and important, especially in the case of video inpainting. Compared to other video applications, any temporary inconsistent case can create a dramatic failure case. See Figure 1. - Temporal consistent results. - In this work, the authors show how the proposed equivariance self-guidance improves video inpainting results and superresolution. - More efficient noise warping. - Compared to the previous work [How I Warped Your Noise], this work improves the inference time by up to 16 times.

Weaknesses

- Better illustration and explanation of how the `self-guidance Equivariance` works. The main novelty of the paper is the self-guidance equivariance, which improves performance compared to previous work, HOW I WARPED YOUR NOISE [1]. However, it is not easy to understand how it works in Figure 2. The current Figure 2 only shows its role and the input and output in the overall system but does not describe its function. Besides, this figure seems to provide more information about how to use optical flow to guide the warping process of noise. - Inpainting examples and usefulness? Do the warped noise and self-guidance Equivariance work for generating "moving objects" in video inpainting? According to the formulation and the generation results, it seems it will favor generating "still objects" in the video inpainting case because it will easily meet the "warped" mechanism. - Experiment Design. Experiments in Section 4.1 are not needed (or do not provide a useful argument) in the paper. Previous works have shown that warped noise is better than independent noise across the frames in a video. The only contribution here is that the fine-tuned version performs better than the un-fine-tuned version. The finding seems unrelated to the paper or provides quite limited information.

Questions

- A better way to illustrate how the proposed Equivariance Self-Guidance works. - A proper figure would be better instead of just formulations. - See Point 2 in Weakness; can the proposed method handle cases where the object is moving or has different actions in the video inpainting case instead of being still?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

For the limitations, the authors mention that in some situations, the work may fail. It would be helpful to also provide the corresponding failing examples, such as the second and third examples mentioned in Appendix F Limitations.