StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

Summary

This paper presents StoryDiffusion, a framework based on diffusion models designed to generate consistent images or videos. StoryDiffusion comprises two key components. The first is a novel self-attention mechanism, named Consistent Self-Attention, which enhances the consistency between generated images and can augment pre-trained diffusion-based text-to-image models without additional training. The second component is a Semantic Motion Predictor module, which predicts motion conditions in semantic space to generate videos with smooth transitions and stable subjects. The combination of these two components in StoryDiffusion offers a lightweight, training-free solution to the problem of content consistency in generated images and videos, while maintaining high controllability over the generated content.

Strengths

1. The article is logically structured and easy to understand. 2. The authors provide clear code examples, making the proposed methods easy to reproduce. 3. The Consistent Self-Attention mechanism can augment pre-trained diffusion-based models without requiring additional training. 4. The Semantic Motion Predictor module extends the method to address video consistency issues, ensuring smooth transitions and stable subjects in generated videos. 5. The paper includes extensive experimental results demonstrating the effectiveness of StoryDiffusion. 6. The framework offers a lightweight and efficient approach to generating consistent visual content.

Weaknesses

1. Although the authors introduce methods like IP-Adapter in the related work and introduction sections, the paper's focus should be on story generation. Therefore, it lacks comparisons with similar story generation works, such as [1], [2], [3]. 2. The paper repeatedly emphasizes that the proposed modules are plug-and-play. However, the main text does not provide corresponding ablation studies to offer more substantial examples. 3. The paper claims that StoryDiffusion is a lightweight method with minimal data and computational cost, but it lacks a detailed analysis of time and space overhead. 4. The quantitative metrics are limited, mostly based on the CLIP score. Additionally, the user study involves a small number of participants and lacks detailed explanations of the setup. [1] Avrahami, Omri, et al. "The Chosen One: Consistent Characters in Text-to-Image Diffusion Models." arXiv preprint arXiv:2311.10093 (2023). [2] Tewel, Yoad, et al. "Training-Free Consistent Text-to-Image Generation." arXiv preprint arXiv:2402.03286 (2024). [3] Jeong, Hyeonho, Gihyun Kwon, and Jong Chul Ye. "Zero-shot generation of coherent storybook from plain text story using diffusion models." arXiv preprint arXiv:2302.03900 (2023).

Questions

The motivation and method introduction are very clear. Here are some experiment-related questions and suggestions corresponding to the weaknesses that need the authors' responses: 1. Could you provide a qualitative comparison with newer story generation tasks such as [1], [2], [3]? These works also rely solely on text prompts rather than input images like IP-Adapter, making them more relevant to your method. Additionally, since [2] also modifies self-attention to maintain content consistency, please explain the differences between Consistent Self-Attention and their approach. 2. Given that the paper repeatedly emphasizes that the proposed modules are plug-and-play, there should be relevant experiments in the main text to substantiate this claim. 3. It would be beneficial to have a clear comparison demonstrating the superiority of StoryDiffusion in terms of inference time and space overhead. 4. For objective metrics, consider including FVD. For the user study, increasing the number of participants would help avoid the collected samples having a cluster bias.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have addressed limitations and societal impacts well. They acknowledge ethical concerns, noting that StoryDiffusion, like other generative methods, could be misused to create false information. They call for clear responsibilities and stronger legal and technical supervision to ensure proper use. In the appendix, they identify two main limitations: minor inconsistencies in subject details, such as clothing, which may require detailed prompts, and challenges in generating very long videos due to difficulties in stitching images with significant differences. Future work will explore these areas further.

Summary

The paper proposes a diffusion-based generative model designed to create subject-consistent images and videos that correspond to given story texts. The paper presents both qualitative and quantitative results showing enhanced subject consistency in generated images and videos compared to existing work.

Strengths

- The paper proposes a training-free and pluggable consistent self-attention module that enhances the consistency of characters in a sequence of generated images. - To effectively model the large movements of characters, the paper proposes a semantic Motion Predictor for predicting the transition between two images in semantic space. - Experimental results show that the proposed model surpasses existing methods in generating subject-consistent images and videos.

Weaknesses

- The Method section lacks clarity: * The process of splitting the story text into several prompts as mentioned in Figure 2 is not described in the main text. * In Section 4.1, the formation of story text from character prompts and activity prompts also lacks clarity. - In L182 and Equation 6, it is unclear whether the text embedding T represent the text associated with frame $F_s$ or $F_e$ or something else. - According to sec 3.2, it seems like the proposed model independently generates transition videos for every two consecutive frames, which could be time-intensive and potentially result in less smooth transitions. - The main paper should include the necessary/essential training details instead of leaving all implementation details in the appendix. - In Figure 6, for the ablation study on sampling rates, it is hard to tell the effect of different sampling rates. It seems for all sampling rates, the model failed to maintain consistency? In the second and third columns, the dog no longer wears the blue vest. Also, according to Algorithm 1, in the actual implementation, the module uses a title size W when sampling the other tokens. However, the value of title size is not specified or discussed. The impact of varying the tile size "W" on consistency is also not discussed.

Questions

- According to Algorithm 1, a tile size W is used for solving the OOM issue. I’m curious whether reducing the sampling rate while maintaining the tile size across the entire batch can be considered an alternative approach? Have the authors experimented with such a configuration? If so, why the current implementation with a specific tile size was preferred? - In L217, how is the CLIP score for character similarity computed? Are any segmentation or bounding boxes applied for computing the Image-to-Image CLIP score? - According to Appendix B, the paper implements upon stable diffusion, which uses VAE for encoding the images into the latents of shape BxHxWxC. In this context, how does the paper define an image token? - In Equation 5, $l$ -> $L$.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitation section has adequately stated most of the limitations. However, the time efficiency of video generation could be discussed more. Generating transitions independently for every two frames could significantly increase generation time. How does this approach compare to other baseline methods like SparseCtrl and SEINE in terms of overall video generation time?

Summary

Aiming to generate a story-based images or videos, this paper introduces StoryDiffusion, and proposes to use the following methods: 1. Consistent self-attention, which is a training-free way that modifies existing self-attention to maintain the between frames. 2. Semantic motion predictor, which is additional module that predicts accurate motions from generating video.

Strengths

- This paper tackles maintaining consistency and similarity between batches, which is necessary when it comes to application to real-world applications. - It maintains consistency in a simple but effective way, with consistent self-attention working without training.

Weaknesses

- The random selection of tokens in consistent self-attention is not well-explained. What are advantages over other baselines despite the randomness? - There are some attention mechanisms that achieve consistency across frames within batches. In particular, some works (e.g., [A], [B], [C], and [D]) heuristically pick frames to attend to (e.g., the first frame, the frame before or after a given frame, or frames in a rotational manner), possibly due to the complexity. It seems that this paper lacks such discussion or comparison. - The novelty seems questionable. It appears to be a post-processing technique or trick for Stable Diffusion that increases consistency between frames. - Typo: L129 has a "." in the middle of the sentence. [A] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [B] Khachatryan, Levon, et al. "Text2video-zero: Text-to-image diffusion models are zero-shot video generators." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [C] Hong, Susung, et al. "DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation." arXiv preprint arXiv:2305.14330 (2023). [D] Geyer, Michal, et al. "Tokenflow: Consistent diffusion features for consistent video editing." arXiv preprint arXiv:2307.10373 (2023).

Questions

1. I'm curious if the consistent self-attention is robust across random seeds, since it picks tokens randomly. Again, it would be great to see comparisons with other token schemes. 2. The Semantic Motion Predictor seems to require video datasets. What is the main motivation for not fully fine-tuning the model with the datasets but only the module?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitations but they haven't discussed the potential social impact.

Summary

This paper proposes story diffusion which is a video generation model encapsulating two contributions: * First, the authors show how to generate a sequence of image that are self-consistent (e.g. consistent appearance for characters, consistent attire, etc) but obey different prompts. This is done using ordinary image diffusion models by generating these images in a batch but having tokens attend to randomly selected tokens in othe rimages in the batch. The authors show that their approach does not require any additional training. * Second, these images (which might be generated following a “story” or sequence of text prompts) can be filled in with higher frequency video frames. The approach is to train a Stable Diffusion model with an AnimateDiff temporal module to use CLIP features of these keyframes (as well as linearly interpolated ones) as control signals. The authors show in evaluations that their approach beats previous papers like IP-Adapter and Photo Maker on character consistency and outperforms SEINE and SparseCtrl on video generation.

Strengths

I very much enjoyed reading this paper. It has strong and compelling results and attacks a timely/important problem. The consistent generation solution is particularly simple conceptually and does not require any training and the second part of the paper builds on top of AnimateDiff2 which content creators should be able to use very easily.

Weaknesses

I do not see any weaknesses that would cause me to vote against acceptance. That being said, the sampling ablation is not very serious (just a handful of cherry picked examples that don’t clearly show any differences relative to the parameter being ablated). More generally it is not so clear why the authors prefer to sample tokens rather than use all of them. Is this for computational reasons? If I am right, the proposed attention should scale linearly in the batch size (so it is unclear why this would be problematic and necessitate sampling).

Questions

* There is some detail provided for the semantic motion predictor in the Appendix that I recommend moving to the main paper (it reads somewhat too abstractly without this detail). * What does plug-and-play / hot-pluggable mean and how is it different from training-free?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes