Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models

Summary

This paper proposed a training-free algorithm to modife cross-attention during inference time that could implicitly minimize the energy in the latent space. This paper formulate this idea from a energy perspective. The authors conduct three experiments - multi-concept generation, image inpainting, compositional editing to demonstrate the effectivenns of the new idea.

Strengths

1. The idea from energy perspective is innovative and the derivation of the equations are solid. 2. Figure 2 an the associate caption is straightforward that shows the relation between energy value and multi-concept generation quality.

Weaknesses

1. In experiment section, the authors provide only some generation cases for subjective-quality analysis without any quantitative results. The shown cases after cherry-pick maybe not adequate to demonstrate the effectiveness of the method proposed in this paper. 2. A minor typo in Line 83: "update rule for a state pattern \epsilon" -> "update rule for a state pattern \zeta"

Questions

1. You metions "nested hierarchy of energy functions" many times? But what is the definition of it and there's any reference papers? 2. In Eq. (6), what is the definition of softmax_1 and softmax_2? And how to obtain Eq. (6) from Q=softmax_2(\betaQK^T)K? I am a little confused.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a novel **energy-based** framework that can automatically **update the context** used in cross-attention **without additional training**. They claim the proposed updating process well solves the **semantic misalignment** issue in text-to-image diffusion models.

Strengths

1. The paper exhibits a clear, engaging, and concise writing style. 2. The paper introduces an interesting energy definition and proves the subtle connections with the attention mechanism used in transformers. 3. The experimental results look promising.

Weaknesses

1. The definition of the energy function does not appear reasonable. For instance, consider the definition of E(K) in equation 8. E(K) increases as the L2-Norm of k_i, where i = 1, 2, ..., N, increases. The minimum value of E(K) is achieved when all k_i converge to zero. This suggests that the author assumes k_i = 0 has the highest probability (since energy is minimum), which contradicts reality. Additionally, the definition of E(Q;K) in equation 7 also seems unconventional. It resembles E(K;Q) rather than E(Q;K). I have significant doubts regarding the effectiveness of the proposed energy definition. I kindly request the authors to provide a more explicit explanation for why they have defined the energy function in such a manner and clarify the actual significance of these proposed energy functions. 2. There is a lack of quantitative experimental results. While the authors present some promising outcomes, there is a scarcity of extensive quantitative experiments to demonstrate that the proposed framework statistically outperforms the naive cross-attention mechanism. It is imperative to include detailed information about the quantitative experiments conducted to validate the efficacy of the proposed framework. 3. There is no explicit explanation of why energy minimization can help solve semantic misalignment. I can not draw direct connections between them in the paper. I tend to reject this paper.

Questions

1. The definition of the energy function does not appear reasonable. 2. Lack of quantitative experimental results. 3. Lack of explanations of why energy minimization help slove semantic misalignment.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

The paper proposes to formulate cross-attention layers using energy-based models such that by minimizing the cross attention energy with respect to the context latent representation, the method can further alleviate semantic misalignments between generated or edited samples and the input descriptions, and allow zero-shot compositional generalization using combinations of cross attention outputs based on multiple inputs.

Strengths

1. The paper introduces a detailed theoretical formulation of cross attentions using energy-based perspective. 2. The method outperforms existing methods qualitatively on various image generation tasks (e.g., inpainting, multi-concept generation and compositional generation) without additional training. 3. The authors have conducted relatively comprehensive evaluations on the method.

Weaknesses

1. **Lack of limitations**. We have seen the good side of such method, which is to improve semantic alignment between generated images and text inputs. However, there are often trade-offs. For example, energy-based optimization is quite unstable, so it requires some tuning in terms of step size and the number of optimization steps. It would be good to mention the limitation of the method, for example, how much the method is slower compared to the standard denoising steps. 2. **Lack of experiments on Image quality**. Though I think only context vectors are optimized, image generation quality will be unlikely to change much. I would still suggest evaluating generated images would provide a way to help understand whether such algorithms can change the generation performance. In addition, although semantic alignment seems to improve in qualitative results, we also need to measure it quantitatively. And current metrics (e.g., CLIP) for measuring semantic alignment can be unreliable, thus a human evaluation seems necessary to compare different methods quantitatively.

Questions

1. is the method quite sensitive to the hyper-parameters that you use to optimize context vectors for each image? Based on the supplementary material, it seems to me that the method is quite sensitive and often requires a lot of tuning in terms of coefficients $\gamma_{attn}, \gamma_{reg}$ for updates, step size, etc.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have included limitations.

Summary

This work tackles the semantic misalignment problem of stable diffusion model using the energy-based model framework.The authors first show that each cross-attention in the diffusion model can be seen as one step optimization of a pre-defined energy function. They then formulate a Bayesian update for the context vector accordingly. The authors demonstrate the effectiveness of their method by showing qualitative samples in multi-concept generation, text-guided image inpainting as well as compositional generation tasks.

Strengths

I love the novelty and theoretical framework of this paper. While there are several other works tackle the more controlable image generation/editing problem based on the modification of the cross attention layers, this paper provides a systematical theory framework for this. This might not only help with the improvement on certain task but also facilicate people to form a deeper understanding of the model itself.

Weaknesses

1. Lack of quantitative results: While several qualitative samples are shown in the paper, quantitative evaluation over a large number of different samples can be more convincing to judge the model's performance against the baseline. The authors may consider using pretrained model (for example like [1] does) or using human evaluation. 2.Choice of hyper-parameter: Looking at the supplementary, it seems that a successful editing requires one to choose parameters $\alpha$, $\beta$, $\gamma$ case-by-case. Furthermore, it seems that changing hyper-parameters have a great influence over the generation performance. This might limit the proposed method to be used in real applications. [1] Training Diffusion Models with Reinforcement Learning

Questions

1. The understanding of the whole unet: While we may look at each individual cross-attention layer as updating a certain energy function, I would like to hear the authors' insight on how we can understand the unet as a whole. At different cross-attention layers, the weight are different. And there are a batch of linear and non-linear transformations between each two cross-attention layers. Then can we understand the whole unet as updating a certrain energy? (The cascading update of C seems to suggest that the optimization across the whole unet has some consistency.) And if possible, how? 2. The energy distribution across different attention layers: In figure 2, the authors plot the energy of 16 attention layers across different sampling steps. It seems that at all the steps, the energy will peak at the middle layer and reach bottom at the two ends. Can the authors provide some insight on why this happens? 3. In section 3.1 algorithm, the authors provide their design of updating context vector C. While this seems to work well on the samples shown in the paper, I'm wondering what if other update algorithms are use. For example, what if we don't cascade $C_{n+1, t}$ to the next (n+1)th layer but updates $C$ for all the layers starting from scratch and update them for more than one step? 4. From figure 8 in the supplementary, it seems that increasing $\gamma_{reg}$ instead of $\gamma_{attn}$ encourages pattern to occur. When $\gamma_{reg}$ is zero, no matter how to tune $\gamma_{attn}$, the editing cannot get successful results (no teddy bear here). This can be a bit counter-intuitive. As for my understanding, the attention term should be the one to align the key and query. Are there any explanations on why these happen?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes an energy-based model (EBM) framework addresses semantic misalignment in text-to-image diffusion models by incorporating EBMs in each cross-attention layer, minimizing a nested hierarchy of energy functions, and achieving highly effective results in diverse image generation tasks. From the shown figures, they demonstrate the effectiveness of the EBM framework.

Strengths

1, The proposed framework demonstrates general applicability to diverse tasks, such as inpainting, image editing, and multi-concept generation. 2, Theoretical analysis and explanations are comprehensively conducted to enhance the robustness of the experimental results. 3, Notably, this paper stands out as the first diffusion model study observed to incorporate an energy-based perspective into its formulation.

Weaknesses

1, The experimental results are only including the qualitative comparison with some other methods. In my view, some quantitative numbers should also be shown in the paper to demonstrate the performance of this method compared to others. 2, For multi-concept generation, the Custom Diffusion [1] has been released during the end of last year, it should be considered as a comparison. 3, The same comparison limitations also happened in inpainting (considering Blended Diffusion[2], GLIDE[3]) and image editing (Prompt2Prompt[4], pix2pix-zero[5]). 4, For evaluation metrics, there are also commonly used metrics in different sub-area. For example, the CLIP-score/Lpips/Structure-Dist can be considered in image editing tasks. [1] Multi-Concept Customization of Text-to-Image Diffusion [2] Blended Diffusion for Text-driven Editing of Natural Images [3] GLIDE: Towards photorealistic image generation and editing with text-guided diffusion model [4] Prompt-to-Prompt Image Editing with Cross Attention Control [5] Zero-shot Image-to-Image Translation

Questions

As I stated above, the main drawback of this paper is in the experimental part. There are lack of qualitative comparisons with correct methods. Instead, the original Stable Diffusion model is always serving as the baseline comparison. Furthermore, there are no sufficient quantitative results to show the efficiency of the proposed framework in this paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

As I stated in the weakness and questions.

Summary

This paper proposes Bayesian Context Update (BCU) and Compositional Averaging of Cross-Attention Output (CACAO). The main idea is to view the cross-attention between text and image as optimizing an energy-based model, and modify the intermediate outputs according to the energy. A series of examples are shown to demonstrate the effectiveness of BCU and CACAO.

Strengths

1. The examples are visually intriguing. 2. The energy viewpoint of the cross-attention mechanism in Stable Diffusion could be enlightening.

Weaknesses

1. I can understand that Q_t is the H * W * C dimensional feature in the network processing the latent-space image at time step t, and C_t is the context vector which is an optimizable variable added by the authors. C_t is initialized to C_clip, then it is optimized going through the layers. As the layers go deeper, C_t is shifted from C_clip, which means the semantic meaning is shifted from the original user text input. My question is: how severe is the shift? How do the authors control the shift? In fact, it makes more sense to me if Q_t,l are the variables to be optimized. 2. More clarification on the examples in Figure 2 is required. I tried Stable Diffusion by myself, and found Stable Diffusion can generate "A monkey and a hat" with the correct semantic meaning. I kindly ask the authors to elaborate how the Stable Diffusion examples are picked. Moreover, Stable Diffusion + Ours degrades the image quality significantly (all the images become black-and-white). Could the authors provide some explanation on why that happens? 3. More quantitative results should be added to prove the effectiveness of the proposed method. In the provided qualitative examples, the advantage of the proposed method over Composable-Diff and Plug-and-Play is marginal. Since the results of generative models are heavily dependent on the random seeds, I suggest the authors to provide systematic quantitative evaluation to benchmark the proposed method. I do notice the quantitative evaluation part in the Appendix, but only animal transition seems limited.

Questions

Please refer to Weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are properly discussed at the end of the paper.

Summary

This paper aims to optimize the context representation through energy based formulation of the cross-attention within the U-Net at test-time to achieve semantic alignment between the textual representation and the image features in the U-Net. Experiments are performed for the multi-concept generation, text-guided inpainting and image editing.

Strengths

+ The motivation of the paper is clear and is supported with adequate experiments. + The approach allows for compositional generation. The qualitative results show that the approach does better than simple DDIM+inversion. + Multiple concepts can be composed together in the generated images with test-time optimization.

Weaknesses

- The formulation of the gradient posterior in Eq. 9 is missing the term on the expectation of the gradient on the right side? - The difference and the motivation to optimize the energy wrt the keys instead of queries is not so clear? Ablations with the two approaches can be done to show the benefits and attention visualizations may help to evaluate the difference between the two. - The subscripts 1, 2 below the softmax are not defined. - In lines 150-151 what is meant by the "forward path of cross-attention"? - Low sample diversity: Example in figure 2 seems to harm the sample diversity wrt Stable diffusion. - What is the computational overhead? How many optimization steps are required to converge to a good solution?

Questions

Please see weaknesses above. Specifically following clarifications are required: 1. The motivation for optimizing keys over queries. 2. The role of the regularization term in Eq. 12. 3. In theorem 1 the updates are wrt to the keys, however, eq 14 considers the energy wrt to the queries. The whole idea of the paper in terms of novelty from theorem 1, then why use eq. 14? The results and comparisons with the baselines can be moved to the main paper. Theorem 1 and Eq 14 need to be better discussed (better combined in one section) and highlighting the contributions and the prior work clearly.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are adequately discussed.

Thank you

The authors conducted additional experiments and clarified the unclear parts. Therefore I increased my score to 5. I suggest the authors to incorporate the new results in the later versions of their paper. Thank you for the efforts!

Thanks for your response

Thanks to your rebuttal reply. I already proceeded to review the supplementary material to check your referenced information. Consequently, I would like to address my final inquiry: in the context of multi-concept composition, are you achieving something similar to Attend-and-Excite [1] framework? Is Attend-and-Excite comparable to your proposed methodology? [1] attend-and-excite: attention-based semantic guidance for text-to-image diffusion models

Thank you for the rebuttal

I have read the rebuttal and the concerns raised have been adequately addressed. That said, the main limitation of the work is in the organization which the authors promise to revise. Taking into account that the peer reviewers are positive that these changes will be made and included in the later versions I will raise my score to 5. I urge the authors to also include a discussion in the main paper or in the supplemental on the computational cost for completeness. Thank you.

Thank you for your rebuttal

The rebuttal solves part of my issues, especially the experimental part. I am excited to see that the author managed to simply manipulate the attention operation via manually defined energy to achieve better generation results without any fine-tuning or relying on any prior. I am willing to accept the paper if all the experiments are reliable. Although, for me, the theory is not that reasonable or at least not smooth or natural. For instance, the author didn't reply about the actual meaning of why they defined $E(K)$ in this form. They only define $E(K)$ to penalize the norm of $Q$. And that seems to be the only reason why they call their method `Bayesian`, which is a kind of patchwork. Considering that the other reviewers all agree to accept the paper, I will upgrade my rate to borderline accept

Thanks for your rebuttal

Dear authors, Thanks very much for your comprehensive rebuttal and discussion for all reviewers. I have carefully read all the discussion and gone through the paper once more. I understand it better, and I think it will be helpful for the image generation community. However, I didn't derive all the equations by myself and read all the reference papers, so I can only give a low confidence (2) and keep my original score (5). Please revise the organization or presentation to make the paper more understandable, as other reviewers suggested.