Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention

Summary

The paper presents a condition-free guidance method for diffusion models. The guidance is generated from the self-attention mechanism to perform guidance from an energy-based perspective as an alternative to classifier-free guidance. With this, the work aims to train the models for improved quality performance in conditional and unconditional image generation.

Strengths

The model presents an alternative guidance method for the diffusion model that is independent of an explicit condition, allowing it to work with conditional and unconditional generation.

Weaknesses

As a suggestion by the paper is to refer to a qualitative assessment, what would be the criteria to consider a model or ablation generates a better quality?

Questions

Questions were mentioned in the weaknesses section. Additionally, it would be recommended to introduce the relationship between reducing the curvature of the enemy function and guidance for context.

Rating

5

Confidence

1

Soundness

2

Presentation

2

Contribution

2

Limitations

The paper discusses potential limitations and societal impact of generative models in the conclusion section.

Summary

The paper proposes a technique to improve unconditional sampling from diffusion models. The main idea is to translate the notion of classifier-free guidance (CFG) to the case in which there is no condition available. To this end, the paper notes that the conditional prediction is "sharp", while the unconditional prediction is "smooth", or more simply put, the unconditional prediction is smoother than the conditional one. The paper also notes that applying Gaussian filtering on the attention weights of the model's unconditional prediction yields an (even) smoother prediction. The paper then combines these two observations, and proposes SEG, a version of CFG that is applicable to the unconditional case: CFG requires the conditional and unconditional predictions; SEG replaces the conditional prediction with the unconditional, while replacing the unconditional with its smoothed version. By doing this procedure, SEG manages to preserve the sharpness-smoothness relation that CFG has between the factors (the conditional and unconditional predictions). The paper then demonstrates that this procedure translates into improved performance compared to reasonable competitors on standard benchmarks.

Strengths

- The paper proposes a simple yet effective procedure to improve unconditional sampling in diffusion models. The idea, in my view, is elegant - The paper is, over all, well written and clear - Both qualitative and quantitative evaluations seem to demonstrate the paper's point on the empirical side - I think the paper's proposal could be used for improving the efficiency of conditional sampling, which is arguably an even more useful case

Weaknesses

## Weaknesses that don't affect my rating - The Method section could strongly benefit from a figure illustrating on what object, exactly, is the Gaussian blur being applied. The paper currently doesn't have a "pipeline" figure - I understand the value of Lemma 3.1, but I don't think it's necessary to provide a proof in the main paper. As a side note, I don't even think it's necessary to provide a proof: as far as I understand, this fact is widely known, since the Gaussian filter is normalized and symmetric, no? - For instance, I think a well-known intuition in the computer vision community is that (infinite) successive applications of Gaussian blurring on an image result in an image with a single color, that corresponds to the average color of the initial image (i.e. same mean, and zero variance). - L188 mentions PAG. It would be useful to remind the reader what it stands for. (I think it was originally just mentioned in the introduction) - Unless I'm misunderstanding, L211 claims that, contrary to two other methods--SAG and PAG--the presented method is both training- and condition-free. I'm not sure I follow: aren't those two methods both also training- and condition-free? - I understand what the authors mean in L231 by "FID has been found not to be a strict measure of image quality". However, FID in itself does not measure image quality, but distance w.r.t. some distribution ------------ ## Weaknesses that don't affect my rating, but should be addressed: - Eq. (5): please consider explaining that "lse" standards for logsumexp a bit before L85: by then it's a bit confusing already - Isn't L86 redundant (since the same statement is made--and referenced--in L75?) - L95: please consider denoting the LSE with some convention: just writing "lse" lends itself to confusion - L97: a negative prediction of what? - L108: to make which operation tractable? - L124: increases the LSE or decreases (according to L127). My intuition is that it decreases, since LSE approximates the maximum of the tensor - The proof of proposition 3.1 could be made shorter, or sent to the appendix - L177: "Note that the Gaussian blur can also be applied to K" I presume this is true because of the commutation property of the convolution operation. Perhaps explicitly stating that in the paper would be better Typos and such: - L9: actually? - L39: why "actually"? - L70: conditioning on - L72: demonstrates its prevalence? sounds weird - L111 and equation at L 113: boldface "a"? - Minor comment for all over the paper: use \eqref for equations - L230: "how much guided images are altered from unguided ones" sounds a bit weird - L232: "being more favorable to users" sounds weird - Fig. 2 and 11: I think there's a repeated sample? (the one indoors)

Questions

- I can understand that the main interest of the paper was improving unconditional generation. The paper achieves this objective by, in a sense, computing a less sharp/more smooth prediction than the unconditional one; this prediction is combined with the original unconditional prediction in the same way that CFG operates. I think that approach is interesting from the point of view of efficiency: a useful prediction is being obtained by an inexpensive operation (i.e. smoothing), instead of another forward pass. Given this context, doesn't it make sense to rather aim at improving conditional generation? (which is arguably the most important one). I think the paper's proposal could be used directly to improve the efficiency conditional generation with CFG: do the conditional forward passes, and replace the unconditional ones with the ones that result from Gaussian smoothing (that is, I think you can replace an entire function evaluation with simple Gaussian smoothing of the conditional prediction). Have the authors considered this option? [I think Eq. (8) with gamma_seg=0 is somewhat similar, but I'd be thinking of replacing s(x,t) with a \tilde{s}(x,t) = smooth(s(x,t,c))]

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, they have

Summary

This paper presents a method for unconditioned image generation based on Diffusion Models, specifically using Stable Diffusion. The proposed method offers an alternative to classifier-free guidance (CFG), eliminating the need to train a classifier for adding conditions. Traditionally, the CFG denoising function incorporates both conditional and unconditional terms, while unconditional generation only involves an unconditional evaluation of the U-Net. Instead, this method performs unconditional diffusion by introducing an additional term to the traditional reverse process. This new term is an energy-based component calculated as a proportion of the Gaussian blurring of the self-attention layers in the U-Net. The core idea is to use a blurred version of the unconditional self-attention layers to shift the mean distribution of the original unconditional prediction, thereby "smoothing" the prediction. The paper employs the standard SDXL model and compares its results against the Perturbed Attention Guidance (PAG) and Self-Attention Guidance (SAG) methods.

Strengths

This paper presents an interesting way to use the unconditional prediction of the network as another way to do guidance generation. This paper presents a method that aims to improve traditional unconditional prediction in diffusion models by smoothing the distribution through the addition of a term to the conventional unconditional image generation process. The proposed idea is analogous to the original CFG methodology in that it combines two different distributions to enhance the generated image. In this case, the smoothing of the distribution is achieved by incorporating a blurred version of the self-attention layers.

Weaknesses

- Technical correctness of the paper The claim in lines 191-192, where the paper asserts that $\sigma \rightarrow 0$ remains the same as the original [image], is not entirely clear, and the intuition behind this is not thoroughly presented. The concern arises because Eq. 7 indicates that the proposed method involves adding another term to the original unconditional generation. According to the literature [10], what this paper refers to as $s_\theta(x,t)$ corresponds to a distribution, as does $\tilde{s_{\theta}}(x,t)$. However, having $\sigma$ approaching to 0 does not imply that this term becomes 0; rather, it means that the Gaussian filter affecting the calculation of the self-attention is 0. Therefore, the inference drawn from this reasoning is that if the filter's variance approaches 0, then $s_\theta(x,t)$ approximates to $\tilde{s}_\theta(x,t)$. Since this paper relies on a strong theoretical background, it is crucial to provide a thorough demonstration and explanation of all claims. For instance, in line 150, what does it mean to be under a "reasonable" assumption to demonstrate the attenuation of the Gaussian curvature? Additionally, what does the paper refer to when discussing the property of linear mapping in the blurring of the queries between lines 167 and 169? As the method uses a Gaussian kernel for the blurring process, it involves a sigma parameter derived from the Gaussian filter. However, there is insufficient discussion on how this parameter significantly affects the saturation of the generated image. From line 211, it can be interpreted that sigma controls the saturation, but between lines 274-276, the paper indicates that saturation can be altered by using the guidance scale for this method. The question is how sigma and gamma can be balanced to avoid over-saturated images and whether other characteristics of the image (brightness, darkness, vibrance, etc.) can be controlled through these parameters. - Experimental validation This paper limits its exploration by not providing further comparisons using other diffusion models as backbones. This omission raises questions about the proposed method's generalization capability. It is not clear why the paper does not report the perceptual metric for the Vanilla SDXL in Table 1. The lack of a metric to compare the perceptual performance of the Vanilla SDXL against the proposed method raises a concern, as the validation is then limited to just a few image examples presented in this manuscript. Without this measure, it is difficult to fully understand the overall performance of the proposed method compared to the baseline and other state-of-the-art methods. The disclaimer in lines 231–232 regarding the FID is acknowledged. However, when it comes to image generation, these are the metrics that the community has adopted and learned to interpret. Of course, this does not limit new research from proposing new ways to measure image generation to provide a "strict measure of image quality" and other attributes. This paper claims (line 14-15) that its implementation does not significantly increase computational cost. However, it is unclear how the computational requirements should be adjusted to use it. Moreover, the paper directly compares its method against unconditional generation, which only requires a single network evaluation, while this method (according to Eq. 7) requires at least two network evaluations. Furthermore, using a conditional input would necessitate at least three network evaluations, likely leading to a considerable increase in computational cost. It is suggested that the paper include an analysis of the trade-off in terms of the number of function evaluations (NFEs) to fully understand how many network evaluations are needed and how this method impacts computational cost. It would have been interesting to see how other filter kernels perform besides Gaussian blurring. - Presentation Although this paper is well-written and presented, its intelligibility could be improved by adding a few more sentences to clarify some of the intuitions described above.

Questions

The questions are listed in the explanation of the weaknesses. It is encouraged to provide further explanation to the presented concerns to have the chance of increasing this initial score. In general, the paper, while presenting theoretical justifications, needs clearer explanations of its claims, particularly concerning Gaussian curvature attenuation and the linear mapping property in blurring queries. The impact of the sigma parameter on image saturation requires more discussion, including balancing sigma and gamma. The lack of comparisons with other diffusion models limits the assessment of the method's generalization, and the absence of perceptual metrics for Vanilla SDXL restricts validation to a few examples. While the FID disclaimer is noted, traditional community metrics should be used.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

NA

Summary

The manuscript introduces SEG, a novel training- and condition-free guidance method for enhancing image generation with diffusion models. The method leverages an energy-based perspective of the self-attention mechanism and introduces a technique to reduce the curvature of the energy landscape of attention, thereby improving the quality of generated images. SEG controls the guidance scale by adjusting a Gaussian kernel parameter, offering a flexible and theoretically grounded approach to unconditional and conditional image generation. The authors validate the effectiveness of SEG through extensive experiments on conditions, showcasing its superiority over existing methods like SAG and PAG in terms of sample quality and reduction of unintended effects. My detailed comments are as follows:

Strengths

This article proposes a novel training- and condition-free image generation method, SEG, which significantly improves diffusion models through the theoretical foundation of smooth energy landscapes and the introduction of Gaussian blur on attention weights and the development of efficient query blur techniques. The quality of the images generated and their superiority verified on multiple conditions.

Weaknesses

The method proposed in this manuscript, SEG, relies heavily on the quality of the baseline model and may amplify biases or harmful stereotypes in existing data.

Questions

(1) The author proposes that the model can generate images without giving any conditions, so how does the model know what to generate? (2) In Figure 2, as σ increases, the image becomes clearer, but the style is completely different from the initial one. Why is this? How to determine the optimal value of σ? (3) What are the main application scenarios of this model? How to generate the image that the user wants without giving any conditions? (4) In Figure 5, Qualitative comparison of SEG with vanilla SDXL, SAG, and PAG. However, the image is labeled as PEG. Please provide a detailed explanation. Also, the full name of PEG is not given. (5) In the manuscript, Smoothed Energy Guidance (SEG) is defined multiple times. Generally, abbreviations should be defined the first time they appear and used consistently in subsequent content. Please also check other abbreviations.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

There is a lack of detailed discussion, such as how to determine the value of σ and why increasing the value of γ does not improve the sample quality in terms of FID and CLIP scores.

Summary

The method (SEG) discussed in the paper mainly applies an energy-based optimization on the emerging values in the self-attention to reduce the curvature of the energy landscape of attention, leading to improved image quality and less structural change from the original prediction compared to previous approaches. SEG is training- and condition-free and can be used for both unconditional and conditional sampling strategies. The authors validate the effectiveness of SEG by evaluating generated images with and without text conditions, as well as with ControlNet.

Strengths

The paper looks at the refinement of image generation through emerging values in attention layers, which is very interesting. The paper investigated multiple conditions for the diffusion model they used, and the visualizations are insightful.

Weaknesses

The quantitative evaluations needed to be presented in a more detailed manner. The number of works they compared against could be much more. e.g. imagic (Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I. and Irani, M., 2023. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6007-6017).), LEdit++ (Brack, M., Friedrich, F., Kornmeier, K., Tsaban, L., Schramowski, P., Kersting, K. and Passos, A., 2024. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8861-8870). ) or collaborative diffusion (Huang, Z., Chan, K.C., Jiang, Y. and Liu, Z., 2023. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6080-6090).),

Questions

What is the advantage of your method compared to other recent works in text-to-image generation that is mentioned in the weaknesses?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The validation can be improved.

Final Questions

Dear Authors, Thank you for your response! The initial comments have clarified the concerns regarding Gaussian curvature attenuation, the linear mapping property in blurring queries, and metrics validation. However, I am still curious about whether this methodology generalizes to other diffusion backbones. Additionally, could you please provide some insight into how other filter kernels perform compared to Gaussian blurring? Best, Reviewer 9qC6

Rensponse to Rebuttal

I thank the authors for the response. I have decided to maintain my score.