DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in orange (color will be removed in the final version). # W1. Whether ICLR is a suitable venue In our method design, we study the feature representation learned by the pre-trained diffusion model to design energy functions. During the editing process, we inject the gradient guidance generated by the energy function into the discrete latent representation of the pre-trained diffusion model, achieving a general image editing framework. Therefore, our method is also a study of the feature representation and latent representation of diffusion models. In addition, we check the Subject Areas of ICLR, which include applications in vision, audio, speech, language, music, etc. Therefore, we believe that our work is suitable for ICLR. # W2. Writing Thank you for these constructive suggestions. We have reviewed the relevant sections and made the necessary modifications. # Q1. Compare with self-guidance Yes, comparing with self-guidance is necessary. Since self-guidance was not open-sourced before the ICLR submission, our original paper did not include a performance comparison with it. Our main difference lies in modeling editing operations. Self-guidance calculates the similarity between the text (corresponding to the object to be edited) and image features, selects regions with high similarity as editing areas, and then applies editing operations. Therefore, the attention in self-guidance is used to locate editing operations. However, this text-guided image editing is coarse. For example: (1) Text can only locate content at the object level, and more accurate local content cannot be expressed in text, making it impossible to be edited with self-guidance. (2) In multi-object scenes, a single text corresponds to multiple objects, making it impossible to use self-guidance to edit a specific object. (3) Text can not provide effective content consistency, leading to deviations between the editing results and the original image. In contrast, our method models editing operations based on the correspondence of intermediate features in the pre-trained diffusion model, which can achieve pixel-to-pixel correspondence, and the editing area is manually defined. Therefore, our method can achieve fine-grained image editing, such as content dragging, which is not supported by self-guidance. In our method, attention is used to inject content consistency guidance, resulting in better consistency between the edited results and the original image. In Sec.4.1 of the updated version, we add the comparison between our method against Self-Guidance in object moving and appearance replacing. We add more comparisons in Appendix.8. These results demonstrate that our method has higher editing accuracy and content consistency.

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in purple (color will be removed in the final version). # W1. The problems targeted in this paper. - drag-style editing is the target task In this paper, we want to build a general framework to complete drag-style image editing, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. This has not been achieved before. - feature matching builds the energy guidance as an editing tool without training In this paper, we choose energy guidance in diffusion models to achieve image editing. Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). Therefore, the task becomes designing a suitable energy function [1] to measure the distance between the current state x_t and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification of $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics in the intermediate features of pre-trained stable diffusion, which transforms the editing target y into the feature matching between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results. [1] Diffusion models beat GANs on image synthesis - memory bank is a container to store information for visual cross-attention Although feature matching can achieve editing operations, the final result can not maintain consistency with the original image, such as in unedited areas. To maintain consistency, we use the memory bank to collect intermediate information during the image inversion and inject it into the editing process through visual cross-attention. This allows the editing process to fully capture the original image information. In Fig.9 of the original paper, we demonstrate the important role of visual cross-attention in maintaining content consistency. Therefore, these components are indispensable for our editing framework. # W2. FID compared with DragGAN This is determined by the base model. DragGAN is designed based on the pre-trained StyleGAN, while our method is based on the pre-trained Stable Diffusion. The generative capability of stable diffusion is significantly better than that of StyleGAN. The results in Fig.8 in the original paper also show that the generative capability of GAN models is limited. For unaligned images, GAN models can produce severe degradation. Even in aligned images, GANs cannot generate a good background, as shown in DragGAN’s results in our paper. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix.7. # W3. Identity by visual cross-attention We need to clarify that our method does not require training. Compared to the latest diffusion-based image editing method, DragDiffusion, our method also has performance advantages without the need for training LORA for each image like DragDiffusion. We will continue to improve content consistency in future work. # W4.&Q1. Ablation study of hyper-parameters and guidance time  Thank you for this useful suggestion. We add an ablation study on the weights ($w_e$, $w_c$) of the image editing ($E_{edit}$) and content consistency ($E_{content}$) energy functions in Appendix 4. Results show that $E_{edit}$ can complete the editing operation with a small $w_e$, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals. The ablation study on when to introduce gradient guidance shows that it is important to introduce guidance in the early sampling stages. As the sampling iterations progress, the effect of the guidance information becomes weaker. Moreover, introducing guidance information in more time steps helps improve the texture details. Finally, we introduce guidance information in the first 30 steps to balance performance and complexity.

Thank you for approving our revision. To address this issue, we explore the editing guidance of our method and DragDiffusion. Specifically, DragDiffusion selects a latent $z_t$ in the diffusion process as a learnable parameter and optimizes $z_t$ through multiple iterations. However, this optimization method only considers the editing target and ignores the impact on the diffusion process, i.e., it cannot guarantee that the optimized $z_t$ conforms to the current time step t. In contrast, our method is built on score-based gradient guidance to solve $p(z_{t-1}|z_{t}, y)$, where $y$ is the editing target. It has been proved that $p(z_{t-1}|z_{t}, y)\sim N(\mu+\eta g, \Sigma)$, where $\eta g$ is the gradient guidance produce from an energy function $p(y|z_t)$. Therefore, our guidance method is compatible with the diffusion process. The energy function (i.e., feature correspondence) in our method is also proven to be accurate and robust in measuring the feature correspondence. We believe that the difference in guidance methods is one of the reasons for the performance difference. We update our paper and demonstrate a comparison of the two editing guidance methods in Appendix 13. We show the editing effects of these two guidance methods without their content consistency design (LORA and visual cross-attention). Retaining only DDIM inversion as the generation prior of the source image. The results demonstrate that the latent optimization in DragDiffusion has a deviation impact on diffusion sampling.

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in green (color will be removed in the final version). # W1. The clarity of how the memory bank is meaningful We need to clarify that the memory bank is a container we designed to store intermediate information ($z_t$, $K_t$, $V_t$) during the DDIM inversion process. These intermediate outputs will be reused to guide the subsequent editing process. In Fig.9 of the original paper, we demonstrated the importance of $K_t$ and $V_t$ (i.e., visual cross-attention) in the memory bank. We also showed the importance of generating content consistency gradients $E_{content}$ through the stored $z_t$. This intermediate information can also be generated in other ways, such as a separate generation branch. But this would result in additional computation cost. In Appendix.2 of the original paper, we conducted an ablation study on the efficiency of the memory bank. However, the connection between these two ablation parts is not tight enough. We added a brief description of Appendix.2 in the ablation study section to highlight the contribution of the memory bank. # W2. How energy design works Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). It has been rigorously proven in [1] for generation control in the diffusion models. To make it work, we need to design an energy function to measure the distance between the current state $x_t$ and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification on $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics of the intermediate features in pre-trained stable diffusion to transform the editing target y into the feature similarity between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results. In Fig. 4, we investigate how to make the energy guidance work better. In Fig. 5, we demonstrate how to combine different energy functions to achieve editing goals. In addition, we update more ablation studies in Appendix.3 and Appendix.4, showing the impact of more components on the working of energy guidance. To make the gradient guidance process clearer, we visualize the gradients generated by the content consistency ($E_{content}$) and editing ($E_{edit}$) energy functions at different time steps in Appendix.5. It presents a gradually converging process, and the gradient generated by each energy function mainly focuses on their respective target areas. [1] Diffusion models beat GANs on image synthesis # W3. Incomparable with DragGAN The higher inference complexity is determined by the base model. DragGAN is designed based on StyleGAN, while our method is based on Stable Diffusion. GAN models only require one inference to output results, while diffusion requires multiple iterations, resulting in a higher time complexity. However, the generative capability of the diffusion models is significantly better than that of the GANs. For example, in terms of robustness, DragGAN requires a specific trained model when editing different categories of images, and the input image cannot deviate from the distribution of GAN's training set. In Fig.8 of the original paper, we show that DragGAN's editing results have severe degradation without image alignment. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix 7. As can be seen, the GAN-based editing method fails in complex scenarios. In contrast, our method is based on the diffusion model with high robustness to edit general images. It is worth noting that the complexity of our method in the preparation stage is significantly lower than that of DragGAN (3.62s vs 52.40s in Tab.1). Most importantly, our DragonDiffusion is a general image editing method that can accomplish a range of image editing tasks, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. In contrast, DragGAN is limited to content dragging.

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in blue (color will be removed in the final version). # W1. More challenging and diverse qualitative samples Thanks for this useful suggestion. We add more content dragging results in complex scenarios in Appendix.7. These results further illustrate that - The generalization ability of DragGAN is limited by the GANs. Each trained model can only edit a specific category of images, such as face, elephant, and car. Moreover, the input image cannot deviate (e.g., unaligned faces, multiple faces or cars) from the training distribution of the corresponding GAN model. Otherwise, obvious degradation will occur. - Our method performs better than DragDiffusion without the need for training. To analyze the performance differences, we further discuss the differences in guidance methods between our approach (score-based guidance) and DragDiffusion (latent optimization) in Appendix.13. The results show that treating $z_t$ as a learnable variable in DragDiffusion can affect the original iteration distribution of the diffusion model. The most important is that both DragGAN and DragDiffusion are designed specifically for content dragging. Our method is a general editing framework, not limited to content dragging. Due to the previous version of the paper focusing too much on content dragging in the experiment, narrowing down this contribution (as pointed out by R1). We have reformulated the experiment section, demonstrating the editing capabilities for various tasks, further highlighting that our approach is a general and fine-grained image editing framework. # W2. Ablation of $S_{global}$ In our method, $S_{global}$ controls the appearance of a certain area to be consistent with a reference area. It is used in the inpainting of the object moving task and the appearance control in the appearance replacing task. Indeed, the previous version lacked an experiment to show the effectiveness of $S_{global}$. We add an ablation study for $S_{global}$ in Appendix.3, demonstrating its appearance control in object moving and appearance replacing. # W3. Ablation of weights for different energy functions Yes, there is a trade-off between different energy functions. The energy functions in our method mainly include $E_{edit}$ and $E_{content}$, designed for editing and content consistency, respectively. We add an ablation study for the weights ($w_e$, $w_c$) of these two functions in Appendix.4. Results show that $E_{edit}$ can complete the editing operation with a small w_e, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals. In our demo video, we directly connect the results of different editing operations into a video, which may amplify some subtle jitter. # W4. Identity preservation  Yes, some details may have some deviations. Our method does not require training, and maintaining high consistency is indeed a challenge. We will continue to improve in future work.

Thank you for your careful review and constructive suggestions. To rectify your concerns, we have modulated our paper. The changes relevant to your concerns are marked in red (color will be removed in the final version). # W1.&W2. More comparisons Thank you for this suggestion to help us highlight the contributions of this paper. A minor clarification is that Self-Guidance was not open-sourced before the ICLR submission, so the initial version did not include a comparison with it. In our new version, we divide Section 4.1 into two parts: content dragging comparison and comparisons on other tasks. In the updated part, we demonstrate our method against Self-Guidance and Paint-by-example in object moving, appearance replacing, and object pasting tasks. We add more comparisons in Appendix.8. Moreover, we further discuss the differences between our method and text-guided editing methods in Appendix.9 by comparing with Null-text inversion, InstructPix2Pix, and Self-Guidance.  - Compare with Self-Guidance. Self-Guidance is a text-guided editing method. Due to the coarse-grained correspondence between text and image content, it can only achieve object-level editing and is difficult to achieve editing objectives in some complex scenes or multi-object scenarios. In addition, Self-Guidance lacks consistency constraints, which leads to deviations between the editing results and the original image. In comparison, the impact of text on our method is minimal, as shown in Appendix.4. It shows that the editing results produced by different texts (relevant or irrelevant) are close and meet the editing requirements. - Compare with Paint-by-example. Paint-by-example is a learnable object pasting method. We must admit that such learnable methods can produce more natural pasting effects. However, encoding the object image into multiple tokens can disrupt the object's identity, leading to changes in the identity of the editing results. In contrast, our method adds guidance at each diffusion step, resulting in better identity consistency in the editing results. - Compare with other text-guided methods (e.g., Null-text inversion, InstructPix2Pix). Although these methods can also achieve object modulation, the editing results have no reference and randomly dependent on text description. In addition, similar to Self-Guidance, text has difficulty achieving a good correspondence with images in complex scenes and multi-object scenarios, leading to editing failures.  It is worth noting that most of these compared methods can only accomplish one task, while we can complete these tasks with a single framework. Through these comparisons, we want to demonstrate that our method is a general fine-grained image editing framework with promising performance. Another related change is that we remove the original Application section and present the application visualization as the teaser on the first page to reduce redundant content. # W3. Typos Thanks for the careful review. We have corrected Equation 2. # W4. Limitations Thanks for this suggestion. We add a discussion of limitations in Appendix.12. It mainly includes that there is still room to further improve the inference speed, and there are some hyperparameters in our training-free pipeline that can affect the editing results.

Thank you for the useful suggestion:) Self-Guidance is a text-guided image editing method, and it is indeed necessary to display the text during its editing process. We have updated the paper (Fig. 7, Fig. 18, Fig. 19) to include the text used in Self-Guidance editing. The words corresponding to the edited objects are marked with yellow boxes. We also find it important to keep the text as simple as possible for Self-Guidance. Otherwise, it will affect the correspondence between the word and the editing object. In comparison, text condition has a minimal impact on our method.