AID: Attention Interpolation of Text-to-Image Diffusion

Summary

This paper proposes a training-free method for generation interpolation of diffusion models with attention manipulation. Targeting on the layout transition inconsistency and nonuniform step-wise transition, the paper proposes to extend the attention interpolation from previous cross-attention to self-attention, and adopts a beta distribution to ease the nonuniform transition. Additionally, three metrics are proposed to evaluate the interpolation qualities, involves consistency, smoothness and fidelity. Experiments show the effectiveness.

Strengths

1. The beta distribution for interpolation weights instead of linear weights seems promising. 2. The proposed metrics seem sound to evaluate the interpolation quality. 3. Experiments show the effectiveness of the proposed attention interpolation.

Weaknesses

1. The application of the generation interpolation seems somewhat restricted, and the practical meaningness of the task is doubted, is the interpolation could benefit other generative applications, or isolated. 2. Basically, the manipulation of the attention map has been broadly explored for image editing, such as [Masactrl; ICCV 2023], [InfEdit; CVPR 2024], etc., and the self-attention corresponding to the layout and the cross-attention corresponding to the semantic are basically common sense, the proposed AID adopts a similar fashion with reformulated interpolation task, where the contribution is somewhat insignificant. 3. The discussion with training-based interpolation methods is suggested to be provided, such as [UniHDA; 2024].

Questions

1. What's the behavior difference between the inner-interpolation and outer-interpolation, is there any guide lines when to use the inner or outer?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately discuss the limitations and broader impacts of their work.

Summary

Summary and Contributions: The authors introduce a novel task called conditional interpolation, which is to generate interpolation images with various conditions like text or pose. They propose an attention-base (and prompt-guided) method to achieve conditional interpolation, and three evaluation metrics to assess the consistency, smoothness, and fidelity of generated images. Extensive experiments shows their method achieve the best performance in image interpolations of diffusion models.

Strengths

1. The propose task(Conditional interpolation) is unexplored and interesting. Comom image interpolation task focus on generate transition images between two real-world images, but limit to one condition. The author introduce both text-embedding and attention mechanism to achieve better performance. 2. The authors do a detail analysis of conditional interpolation, specifically text-embedding interpolation. They prove that text-embedding interpolation is equivalent to manipulating the keys and values in cross-attention module and find that doing similar operation in self-attention layer can significantly improve spatial consistency. 3. The method can be used in image interpolation with distinct conditions like “a truck” and “a cat”. The authors also show their method can improve image-editing results when use editing methods like p2p. 4. The authors introduce a third prompt to guide the interpolation between two prompts, which is interesting and useful. Correctness: the claims and method are correct. Clarity: The paper is well written Relation to Prior Work: The paper is clearly discussed how their work differs from previous contributions.

Weaknesses

1.The paper lack of a clear and detail definition of the conditional interpolation. The authors claims that they formulate a new task call conditional interpolation, which is doing interpolation under various condition, such as text and pose. But I am confused that the method they proposed only condition on text, how could it be various condition? What the definition about various condition? 2.The comparations between baselines are inconsistent. The table 1 shows the result of TEI and DI, but the table 2 only show TEI. 3.The qualitatively compare between baselines（TEI and DI）is lack, while the quantitative result of this two baseline are exist.

Questions

No coded is provided, which I think is important.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

see weaknesses.

Summary

In this work, the authors propose Attention Interpolation via Diffusion (AID), a novel, training-free technique for improving image interpolation under specific conditions like text or pose. Traditional methods using linear interpolation often produce inconsistent, low-fidelity images. AID enhances image consistency and fidelity with a fused interpolated attention layer and selects interpolation coefficients using a beta distribution for smoother results. An advanced variant, Prompt-guided Attention Interpolation via Diffusion (PAID), treats interpolation as a condition-dependent generative process. The authors include the experiments to demonstrate AID's consistency, smoothness, and efficiency in condition-based interpolation. The work also includes user study to show AID better aligns with human preferences and aids compositional generation and image editing control.

Strengths

1. The work includes user study to compare performance of different models which is appreciated. 2. The work is motivated to improve existing image interpolation methods based on their drawbacks.

Weaknesses

1. The work proposes a few metrics to evaluate the quality of interpolated images. However, these metrics could be biased and fail to evaluate the quality of samples. More details are discussed in Questions. 2. Though image interpolation can generate interesting visual effects. I feel the actual applications for such technique could be limited.

Questions

1. For the fidelity metric in Eq. 7, one should expect the metric is maximized when the interpolation starts the same as image A, has an abrupt change from image A to B, and stays as image B for the remaining. This could be against the objective to generate perceptually consistent and smooth images. 2. Based on proposition 1, text embedding interpolation can be viewed as manipulating the K/V which leads to suboptimal interpolation. However in (fused) inner-interpolation, still only K/V are manipulated. 3. Is there a systematic way to determine the coefficient for Beta distribution, or is it a hyperparameter that needs tune for different models or prompts? 4. How does applying interpolation module in self-attention or cross-attention alone affect the performance?

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The work sufficiently addressed the limitations.

Summary

In this paper the authors explore the task of interpolation between images in conditional diffusion modal. First the authors list out the three desirable properties of successful interpolation: perceptual consistency, smoothness, and image quality. The authors first introduce a method AID that incorporates three ideas: (1) interpolation should be done with both cross attention and self attention, (2) instead of a simple interpolation a fused interpolation should be performed, (3) the interpolation coefficients are sampled with a beta distribution instead of sampling uniformly. The authors also introduce a second variation of their method called PAID (Prompt guided conditional interpolation) where the users can optionally add a prompt representing the intermediate image.

Strengths

- The paper is well written and well organized. The authors do a good job first analyzing the issues with interpolating just the text prompt and then proposing a method that addresses them. - The visual results shown in the paper look impressive and the authors show that the method can also be applied to other tasks like image editing and compositional generation. - The authors use a comprehensive set of metrics that capture the 3 different aspects of good interpolation and show that the proposed method is helpful for each of the three metrics.

Weaknesses

- One aspect of diffusion models that the authors have not considered here is the classifier free guidance and the use of negative prompts, which is standard practice for generating high quality images. Appendix H in supplement mentions that there are some results shown with negative prompt. But it would be useful to have a more detailed discussion of this in the main paper. - The authors show an ablation study in Table 1(b). However the discussion for these experiments is very brief. It would be useful if the authors could show some visual results corresponding to these ablation experiments and a more thorough discussion. (Minor): - “text-to-diffusion” → “text-to-image” L94

Questions

Please see the weaknesses above.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have included a limitations section in the paper.

Official Comment by Reviewer frah

Thanks for the authors rebuttal. Basically, the mentioned image editing and compositional generation refer to the feature interpolation and prompt interpolation, and the proposed AID could provide the strength adjustmnet, which may still somewhat limited in application, and what the user can control is quite less. As the main concern is still exist, I maintain my score.