Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

Summary

This paper argue that I2V-DMs tend to overly rely on the conditional image at large time steps, resulting in videos with less dynamic motion. To address this, they propose using the KL divergence between the initial noise distribution and the actual marginal distribution to enhance the amplitude of motion while preserving video quality. Besides，the paper propose a time-dependent noise distribution for the conditional image during training process.

Strengths

Motivation. The challenge of conditional image leakage presented in the paper is a credible issue and represents a significant academic problem. The paper is exceptionally well-organized, making complex concepts easily digestible for the reader.

Weaknesses

1. Claim. Regarding the conditional image and the diversity of content changes (referred to as 'motion' in this paper), it is, in fact, a matter of trade-off. The essence of image to video generation is to leverage the conditional image to enhance video quality. Therefore, the Proposition 1 and Training Strategy proposed by the authors are essentially fine-tuning techniques that favor the diversity of motion. 2. Additionally, the foundation model of video generation guided only by the first frame inherently contains uncertainty in its output; it does not know whether the user wants to generate a video with significant or minimal motion. Some recent studies (last frame guided[1], motion control[2])have demonstrated that incorporating additional guidance signals can indeed achieve big and precise motion. Particularly, the works DragNUWA and MotionCtrl have shown that for the image-to-video generation task, it is still possible to generate significant motion. Therefore, the foundation model of SVD has the potential to directly generate substantial motion. In this context, the initial version of SVD, which could only generate minor motion, might be due to the lack of a specific condition or guidance signal for significant motion, rather than an inherent incapability. From a subjective standpoint, not all applications require substantial motion. [1] ToonCrafter: Generative Cartoon Interpolation [2] MotionCtrl: A Unified and Flexible Motion Controller for Video Generation [3] DragAnything: Motion Control for Anything using Entity Representation [4] DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory 3. Experiments. The experiments appear to be somewhat confusing and not entirely convincing. Regarding the variables of the related training strategy displayed in Figure 8, the reviewer has not found the corresponding ablation study, which is a core contribution of this paper. Additionally, the author has not provided an ablation study on the 'M' of Proposition 1, such as testing performance changes between 0.6T and 1T, which is central to the first contribution. If the reviewer has overlooked it, please point it out. Additionally, Figure 9 also appears to be somewhat confusing. Was a user study not conducted on the Analytic-Init? What does '-tune' signify—does it mean finetuning (training strategy)? If so, why does it appear in the inference phase? -------------------- After reading the author’s and other reviewers' feedback, I have adjusted my score to 5.

Questions

Can the authors provide more insights into the selection of hyperparameters (Line 230 Implementation Details) for the time-dependent noise distribution?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

yes, the authors have discussed the limitations.

Summary

The paper points out that existing image-to-video diffusion models can lead to videos without significant/desired amount of motion. Some evidence is presented to show that this is due to what the authors call "conditional image leakage" where the model places too much emphasis on the conditional image and all the generated frames look too similar to the conditional image. Two strategies are proposed to overcome this issue. The inference strategy starts the reverse diffusion process at an earlier time and uses improved initial noise distribution parameters using a closed-form expression to match the noise parameters at training time. The training strategy develops a time-dependent noise distribution to have higher noise at larger t to place lesser emphasis on the conditioning image for larger t values. Experiments indicate improved video generation using either or both of these strategies.

Strengths

1. The methods presented in the paper are well-motivated and make intuitive sense (although there appear to be some technical issues as noted in the weaknesses below). 2. Results: The quantitative results show clear improvements over baseline. The video examples submitted also show clear improvements over previous methods 3. Relationship to existing works: the authors have done a good job presenting related works carefully and how existing methods may also inherently overcome some issues, if not all.

Weaknesses

1. Issues with derivation: For deriving the expression of better noise initialization, the authors present their proof in Appendix A. The second and third terms of (7) are considered constants for the purpose of optimizing the noise distribution. I don't follow this and think there may be a technical error here. It is written that since $q_M(X_M)$ and $p_M(X_M)$ are independent, the entropy $H(\mathcal{N}(X_M; \mathbf{\mu}_q, \Sigma_q))$ and $H(q_M(X_M))$ can be considered constants for optimizing the parameters $\mathbf{\mu}_q$ and $\Sigma_q$. I don't think this is correct as these parameters influence both $H(\mathcal{N}(X_M; \mathbf{\mu}_q, \Sigma_q))$ and $H(q_M(X_M))$. Can the authors please explain this in detail? 2. Quantifying motion: The authors say that with previous works the models could in principle just output the conditioning image several times and still lead to lower error or improved quality metrics like FID and IS. However, they still use the same metrics for their experiments. Perhaps quantifying amount of motion or even better, quantifying realism of motion directly is important. 3. Figure 9 is not clear. What are orange and blue colors indicating. Please explain this figure in detail and how it shows that the proposed method is considered better by the user study. I see very little differences going from <method> to <method>-tune in the figure.

Questions

No additional questions.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes, authors have addressed some limitations of their method.

Summary

This paper investigates and proposes solutions for the problem of “conditional image leakage” in image-to-video generation. The authors claim that existing image-to-video diffusion models over-rely on the conditional image at large diffusion timesteps when the inputs are too noisy, leading to static video output. To address this issue, the authors propose an inference-time generation process that skips those large timesteps. The authors also propose adding noise to the conditional image during training to mitigate conditional image leakage.

Strengths

This paper identifies the problem of conditional image leakage and shows that existing methods suffer from static motion at large diffusion timesteps. The two proposed solutions empirically boost FVD and IS on UCF101 and bring significant improvements to motion magnitude while maintaining a similar image alignment and temporal consistency in the user study. These show that the proposed approach is indeed useful in generating more dynamic videos.

Weaknesses

There are a lot of prior works [a, b, c] that deal with noise initialization in video diffusion models that are highly related to the problem of conditional image leakage and the proposed inference technique. There are, however, no discussions on the differences between these techniques. Notably, FreeInit [a] and FrameInit [c] perverse the low-frequency component of the initial image – which is at odds with the proposed theory. I believe an in-depth discussion and a comparison with these approaches would help to establish the thesis of this paper. Additionally, PIA [d] also allows motion control – comparing the proposed motion-adding techniques with PIA’s motion control would also be helpful. [a] FreeInit: Bridging Initialization Gap in Video Diffusion Models [b] Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models [c] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation [d] PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models Secondly, I am not entirely convinced by the authors’ explanation of conditional image leakage. The authors claim that models learn a shortcut by copying the initial image at large diffusion timesteps, leading to static video output. However, models like VideoCrafter and SVD do produce outputs of diverse motion of some degree, as shown in their papers. What are the authors’ explanations for these motions? Where are they coming from? Trying to copy the conditional image when the input is otherwise noise is likely the optimal solution and would be learned – but that is not a sufficient condition for static output. Thirdly, how are evaluations normalized for motion magnitude? Methods like SVD and PIA have motion control. Since the evaluation is focused on “dynamic motion”, perhaps choosing a different motion parameter for these methods would lead to different results in the user study. How are those parameters chosen? Do baseline SVD, SVD after finetuning, and SVD with the inference trick interpret the same motion scale parameter in the same way? One way to validate this is to plot the input motion scale parameter against the output motion magnitude (e.g., optical flow) for parameter selection.

Questions

What are the motion magnitudes of SVD in Figure 3 and Figure 7? Adding different noises to the conditioning image at different time steps requires the conditioning image to be embedded multiple times. How does this affect inference time?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper presents an approach for image-to-video (I2V) generation. The authors start with a conditional image leakage problem in existing works where the conditional image significantly influences the generation process thereby impacting the dynamism of the video. The authors propose an inference time and a training time strategy to mitigate the problem. The approach is evaluated on UCF101 and the authors' ImageBench dataset. The results are impressive.

Strengths

1. The authors identified a crucial problem of conditional image leakage with I2V generation.  2. The authors proposed two strategies to address conditional leakage. An inference time strategy that can be readily applied to existing approaches. Another approach is applied during the training time. 3. The paper is well-organized and easy to follow.  4. Authors conducted a user study for qualitative evaluation. The results are impressive.

Weaknesses

1. The authors are encouraged to provide more details about the ImageBench dataset. 2. While performing the user study, what qualities are evaluated by the users? 3. How is the accuracy of the generated videos estimated corresponding to the text prompt?  Do the users rate the matching score of the generated video corresponding to the text prompt? 4. The realism of the generated videos is lacking. This is common for diffusion-based generated visual content.  Have the authors considered how to improve the realism of generated videos?

Questions

Please address the comments in the weaknesses section.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Looking forward to further feedback

Dear AC and Reviewers, Thank you again for the great efforts and valuable comments. We have carefully addressed the main concerns in detail. We hope you might find the response satisfactory. As the discussion phase is about to close, we are very much looking forward to hearing from you about any further feedback. We will be very happy to clarify any further concerns (if any). Best, Authors

Thank you for your response

I thank the authors for their response. I under the derivation better now, I think misunderstood some parts initially. The additional results quantifying motion could be of value in the final paper, if accepted. And they are appreciated. Overall, the authors have addressed my concerns, and I believe they have addressed most of the other reviewers' concerns. I am raising my score to a 6 based on NeurIPS rating scale.