Summary
This paper aims to understand two mechanisms of diffusion models. First, the denoising process is analyzed, and it is found that shapes in an image are constructed in the beginning of the denoising process, while textures and details are filled in later. This empirical observation is justified with a mathematical frequency analysis. Second, the role of text conditioning is analyzed and it is found that the [EOS] token, which captures global information of the prompt, is relied on more heavily by the diffusion model. It is also observed that the text prompt is utilized more in the earlier stages of the denoising process. This finding is utilized to speed up diffusion sampling by ~25% while maintaining the image quality and prompt alignment. This is done by only injecting conditional information in the beginning of the denoising process.
Strengths
* Although the finding that shape is constructed in the first few timesteps has been observed many times before, it is nice to have a more principled study with various experiments and mathematical justification.
* The finding that the special [EOS] token is the most relied upon during generation rather than the prompt tokens is an interesting finding that can be used in later studies. For instance, improving prompt alignment, attribute binding, etc.
* The observation that the text prompt is used more in the early denoising process lends itself to a practical application of speeding up inference.
* Multiple architectures and samplers are used in this study, suggesting the generality of these findings.
Weaknesses
* As mentioned in the Strengths section above, the findings are not completely surprising (for instance, the shape reconstruction or reliance on text in the early denoising steps, then detail-filling in the later steps). However, this work takes a principled approach in studying these phenomena which have largely been used in diffusion application literature (e.g., [1, 2])
* Limited to no mention of broader impact or limitations. Furthermore, the Conclusion section is just a summary of the paper but does not discuss the implications of these findings.
[1] @inproceedings{mengsdedit,
title={SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations},
author={Meng, Chenlin and He, Yutong and Song, Yang and Song, Jiaming and Wu, Jiajun and Zhu, Jun-Yan and Ermon, Stefano},
booktitle={International Conference on Learning Representations}
}
[2] @inproceedings{hertzprompt,
title={Prompt-to-Prompt Image Editing with Cross-Attention Control},
author={Hertz, Amir and Mokady, Ron and Tenenbaum, Jay and Aberman, Kfir and Pritch, Yael and Cohen-or, Daniel},
booktitle={International Conference on Learning Representations}
}
Questions
* What are some of the limitations and implications of these findings?
* I did not take this into account for my review, but there are many typos in the text and figures which can be corrected for the next version.
* This (https://arxiv.org/pdf/2404.07724) is a concurrent work, so it is not expected for this paper to compare against it. Their finding is applying guidance in the middle denoising steps improves image quality and distribution coverage. I am curious to hear how the findings this paper being reviewed can be connected to the phenomena observed there. It might be something to keep in mind for the next version, although it will not be used in assessing this paper.
Limitations
Although there are no societal implications, a discussion of limitations is lacking.