Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing

Summary

This paper proposes a dynamic prompt learning approach for image editing that modifies self-attentions to more accurately attend to the correct nouns given a text prompt. The proposed approach is used along with null text inversion where the dynamic tokens corresponding with the noun words are updated with a background leakage loss, a disjoint object attention loss and an attention balancing loss. Then p2p is used with the modified cross-attention masks to edit the image. The main benefit of this approach is in editing an image which includes two or more semantically related objects like cat and dog. The authors have collected a set of 60 multi-object images from LAION as a test set and a set of 60 images with semantically related objects for their human evaluations, have shown improvements through qualitative and some quantitative evaluations.

Strengths

1. This paper is well written and explains the method and related works clearly. 2. A novel method as dynamic prompt learning for image editing has been proposed. The paper is mainly focused on a specific axis of image editing where two or more semantically related objects are present in the image and cross attention maps for each object leak to other objects or the background. This approach modifies the attention masks and thus can be used along with p2p for image editing. 3. A test set containing images of multiple objects from LAION-5B is collected for quantitative and qualitative evaluations and a couple of examples are presented in the paper and supplemental showing the superiority of the proposed method.

Weaknesses

1. The scope of the image editing problems this paper aims to improve is narrow mainly useful for cases when there are two or multiple semantically-related objects in the image and for word-swapping editing prompts like cat to dog, cat to tiger, etc. 2. Through only a couple of image editing examples shown in the paper, I am not convinced that this paper is beneficial in general image editing problems where there are not necessarily related objects in the scene or for other types of editing prompts rather than word swapping. - It is unclear what type of editing prompts are included in the collected test set, but from discussions, it looks like it is mainly focusing on word swapping. Also, size of the test set is fairly small, and the studies are all done in the above narrowed down scope in image editing, thus not conclusive. - The quantitative clip scores are only reported in two scenarios (1) cat → leopard and dog → tiger and (2) book → box and clock → sunflower and improvements are still marginal. - On the human evaluation study, I'd expect it to be done on other image editing types with single or multiple objects either semantically related or unrelated, on a larger set, and to compare with other baselines such as instructpix2pix and plug-and-play. 3. On auto evaluations, one potential metric to measure the correctness/accuracy of this method could be applying an object segmentation or object detection method on the edited image, predict label for the edited objects, check if the predicted label matches with the given prompt. Compute the average accuracy score and compare with the baselines.

Questions

1. Please describe your evaluation datasets and their editing prompts more clearly. Is the URS-Set similar to the MO-Set? what is the difference between these two sets? Since LAION-5B is pretty large, I'd suggest the authors to expand their test set to include more varied image editing prompts and examples. 2. I wonder how much gain this method introduces on image editing problems with one object or two unrelated objects in the image (if any).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have neither discussed the limitations of their work, nor its negative societal impact. A broader study on different editing types as mentioned above could clarify the limitations.

Summary

The paper proposes a new method to improve the attention masking for attention-based local editing.

Strengths

The proposed loss functions are novel, intuitive and effectively improves the inversion process of text-to-image diffusion model.

Weaknesses

1. Please show the quantitative evaluation of the object masking performance between the mask generated from Diffedit. 2. Quantitative comparison with baseline models are not enough. Please put more comparison between other baselines such as Plug-and-play and Pix2pix-zero, instructPix2Pix. At least include user study. 3. Does it improve other attention-based editing mechanism?

Questions

Please see the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No

Summary

The authors first point out that inferior cross-attention maps regarding noun text tokens are the main causes of failures cases in prompt-based editing methods, such as prompt-to-prompt (P2P). To tackle this, they propose to optimize noun text features with three objectives at each denoising timestep; i) minimize the overlap between different cross-attention maps, ii) keep the background mask intact, which is automatically inferred from both cross- and self-attention maps, and iii) keep the max value of cross-attention maps related to nouns high. This way, they achieve better editing results with cross-attention maps that are better aligned with the object regions.

Strengths

- the authors tackle a meaningful task - it is easy to understand the motivation and method with various visualizations - the proposed method presents improved performance on a variety of evaluation metrics

Weaknesses

- increased inference time - only can solve cases related to noun text tokens representing the type of object

Questions

- regarding the algorithm 1, isn’t it better to use updated text features by implementing the line 9 after the line 13? - the thresholds defined by Equation 10 appear to prevent overfitting of the model. what happens if you don't use these thresholds?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

please refer to the Weaknesses section

Summary

The paper propose DPL to solve the cross-attention background and distractor object leakage problem in image editing using text-to-image diffusion models. The presentation is well written and easy to follow. The discussion and analysis is extensive and interesting. But the experiment dataset is too small to clarify the robustness of this method.

Strengths

1. The presentation is well written and easy to follow. 2. The discussion and analysis is extensive and interesting. 3. The framwork to solve the cross-attention background and distractor object leakage problem is novel.

Weaknesses

1. The experiment dataset is too small to clarify the robustness of this method.

Questions

null

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

null

Summary

This paper proposes Dynamic Prompt Learning (DPL) to address the cross-attention leakage issue for text-based image editing. Based on the observation that inaccurate cross-attention maps cause unintended modifications of regions outside of the targeted area for text-based image editing, the authors propose Dynamic Prompt Learning to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, the proposed approach achieves fine-grained image editing over particular objects while preventing undesired changes to other image regions. Experiments of Stable Diffusion models are conducted for word-swap, prompt refinement, and attention re-weighting.

Strengths

1. This paper proposes an interesting approach to address the attention leakage problem for text-based image editing. By adding constraints on the cross-attention maps, the proposed approach regularizes the attention regions of the diffusion models and thus achieves detail-preserving and high-fidelity image editing. 2. The proposed approach is well-motivated and the discovery of the attention leakage problem is interesting and inspiring. 3. The proposed approach achieves better text-based image editing performance on various editing scenarios, as demonstrated in the extensive experiments.

Weaknesses

1. The proposed DPL approach seems like a combination of several components (including the Disjoint Object Attention Loss, Background Leakage Loss, Attention Balancing Loss, Gradual Optimization for Token Updates, and the trick from null-text inversion), and it is not clear about the contribution and necessity of each component. More ablation study is needed to demonstrate the necessity of combining those methods and loss functions. 2. It is not shown how accurate and how robust the attention-based background estimation is. When does the background estimation fail and how this will affect the model if this part fails? The choice of hyperparameters (self-attention with size 32, cross-attention with size 16 and TH = 0.2, V = 5) for this subsection is based on empirical study, and ablation study is not presented. 3. The writing can be improved. The structure of section 3 can be organized more logically and more clearly. The paragraph "Gradual Optimization for Token Updates" is unclear to me. 4. Experiments are conducted on a self-built dataset with a small size (only 100 images for ablation study and 60 images for user study).

Questions

Please refer to the weakness section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors addressed the limitations adequately.

Summary

This paper works on the fidelity problem (i.e., 'unintended changes of background and distractor objects') in text-based image editing of text-to-image diffusion models. The authors attribute the fidelity problem to cross-attention leakage and propose dynamic prompt learning to force cross-attention maps to focus on correct noun words in the text prompt. The paper shows a comparison with a couple of approaches, comprehensive image editing evaluations and ablations.

Strengths

1. The fidelity problem that this work focuses on is important. The overall idea of the paper is really simple and seems effective to this problem. 2. The authors present a good number of experiments validating the effectiveness of their approach. The results from the paper are promising. The editing quality is decent with observable changes. 3. The paper is overall well-written and easy to follow.

Weaknesses

1. This work has the following potential limitations: - The attention map limits the size of the generated target, as shown in Figure 7. - Not applicable to the scenarios in that the noun describes the background, or changes are reflected in the verb word, such as adding an item or deleting objectives. Thus, more experiments in the above scenarios should be provided and discussed. - Not training-free and may have high latency in practical applications. 2. Both qualitative and quantitative results miss comparison with important methods such as DiffEdit [1], Imagic [2], StructureDiffusion [3]. 3. The manuscript lacks explanations for some symbols, such as symbols in Equation (6). It would be great if the author could highlight these symbols in the diagram to refine the explanation of the method. [1] Diffedit: Diffusion based semantic image editing with mask guidance. ICLR 2023. [2] Imagic: Text-based real image editing with diffusion models. CVPR 2023. [3] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023.

Questions

1. Please clarify the similarities and differences between DPL and Imagic [2]. In my opinion, both are looking for optimized text embedding, while Imagic seems to remain open to control objectives, while DPL adds too many restrictions through attention maps. 2. How long do training and inference take in the proposed method? Can the authors detail the advantages of DPL compared to the training-free methods (e.g., StructureDiffusion [3])? 3. What are the dimensions of the attention maps, the query/key matrix, and the deep features of the noisy image. It will be great if the authors could indicate those dimensions in the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have provided discussion on the limitations and broader impact in the Supplementary Material. However, as mentioned in the weaknesses, this work has the following potential limitations: - 1) The attention map limits the size of the generated target. - 2) Not applicable to the scenarios that the noun describes the background, or changes are reflected in verb word, such as adding an item or deleting objectives. - 3) Not training-free and may have high latency in practical applications.