Conditional Score Guidance for Text-Driven Image-to-Image Translation

Summary

This paper introduces a novel approach for text-driven image-to-image translation tasks. The main contribution of this work is the development of a conditional score function that takes into account both the source image and text, in addition to the standard condition with the target text. The new score function consists of two terms: the standard score function conditioned on the target prompt, and the guiding score function, which models the posterior of the source latent given the target latent and target prompt. The paper also proposes an effective mixup strategy in cross-attention layers of the text-to-image diffusion model to facilitate image-to-image translation. Experimental results that demonstrate the outstanding performance of the proposed method on various tasks. The paper also introduces an intuitive performance evaluation metric that measures the fidelity of pairwise relations between images before and after translation.

Strengths

1: Detailed theoretical explanation for the algorithm used and the overall setup, providing a strong foundation for the study. 2: The paper offers a novel mixup method that enhances the conditional score guidance, and the mixup method effectively combines two outputs of cross-attention layers. 3: Comprehensive experimentation and qualitative/quantitative measures back up the claims. Comparison with state-of-the-art methods and showcased superiority in most cases. 4: The paper provides a novel metric (Relational Distance) for evaluating the methodology. The new metric quantifies how faithfully the relational information between source images is preserved between translated target image, which is a good contribution.

Weaknesses

1: Lack of clarity on the practicality of the proposed method. It would be very valuable to discuss the computational cost (time, memory) and efficiency of the proposed method compared to other techniques. 2: The paper only focuses on one pre-trained model (Stable Diffusion) for the experiments. More experiments regarding other pre-trained methods should be shown to show the proposed method generalize well. 3: Insufficient coverage of the limitations of the method. There is no extensive discussion on the limitations of their method and what scenarios it won't work well in

Questions

The method proposed appears to be heavily reliant on the quality of the source image. Could there be issues if the source image isn't high quality, for instance, leading to degrading the results? Please see the weaknesses for more. I will change my rating based on the rebuttal and other reviewers' comments.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are not discussed. One potential limitation is listed in the questions.

Summary

This paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.

Strengths

This paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.

Weaknesses

Incremental Novelty: Although the paper proposes conditional scores that replace the original scores of a pre-trained text-to-image diffusion model for image-to-image translation, the core methodology appears to be significantly built upon pix2pix-zero. To elevate the novelty, it's recommended that the authors shed more light on how their method extends beyond pix2pix-zero's capabilities. This can be achieved by emphasizing the unique aspects of the approach and providing a clearer delineation of the differences from pix2pix-zero. In addition, regarding performance, the qualitative results provided in the Appendix do not consistently demonstrate superior performance compared to pix2pix-zero. Thus, to substantiate the claimed superiority of the proposed method, it would be beneficial to include additional examples, including those not addressed by pix2pix-zero, to showcase the broader effectiveness of the paper. Limited Editing Capabilities: The proposed method seems to primarily focus on object-centric editing. The paper could benefit from demonstrating its capabilities for more precise or finer edits to articulate the efficacy of the proposed components, particularly their role in selectively editing the region of interest while retaining irrelevant parts of the image. Lack of Comparison: The paper should discuss with concurrent work, Delta Denoising Score [1] which was introduced as same purpose as CSG to edit images with minimal modifications. Furthermore, a performance and methodological comparison with Plug-and-Play [2] is highly recommended to provide a more comprehensive understanding of image-to-image translation. Missing parts: The authors have not provided any discussion on the limitations and potential negative societal impacts of their proposed model, a crucial aspect that is currently missing. [1] Hertz et al., Delta Denoising Score [2] Tumanyan et al., Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

Questions

See the weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Since the paper considers image-editing, the author should demonstrate the case when the proposed method fails, e.g., due to the incompetence of pretrained text-to-image diffusion models, the complex prompt might fail, or some attributes that are hard to edit. Also, some ethical warning must be included in the paper to demonstrate the possible misusage of image editing methods.

Summary

The authors are proposing two sampling techniques in Diffusion Models; Cross Attention Mixup and conditional score guidance.Experiments show that the proposed methods show decent performance compared to baselines.

Strengths

- Qualitative results are interesting - Reasonable motivations and methods

Weaknesses

Weaknesses are written in the Question part below.

Questions

* (Important) How can we get Eq. 8 from Eq. 7? I understand the second term ($p(x_t^{src}|…)$) in integral ends up being one. My question here is that how can the integral on the first term ($p(x_t^{tgt}|\hat{x}_t^{src}…)$) be removed (or ignored) to get Eq. 8? The simplest way would be to think of it as a constant w.r.t. $x_t^{src}$. However, it is not the case since the conditional (the first term in Eq. 7) would be equal to $p(x_t^{tgt},x_t^{src}|y^{tgt})p(x_t^{src}|y^{tgt})$ by Bayes rule, and it is not a constant w.r.t. $x_t^{src}$. * (Line 35-36) A bridging paragraph is needed. * (typo, Algorithm 1) a target prompt embedding. * It is not intuitive how the covariance $\Omega^{-1}$ is computed, and what role it plays. Can it be visualized? e.g., use only the posterior guidance when do sampling and directly visualize Eq. 14. * It seems like $\Omega \in \mathbb{R}^{H \times W}$ while diagonal covariance matrix needs to be in $\mathbb{R}^{C \times H \times W}$ (considering the diagonal term only) * What about sampling speed w/wo the additional guidance term? * (Important) * CSG (wo mixup) evaluation: * Classifier free guidance (best weight needs to be searched, e.g., 3-7) v.s. CSG (wo mixup) * Ablation: * DDIM + mixup v.s. DDIM => to show the effect of mixup

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Additional experiments are needed to validate each of proposed method separately. - Additional comparison between the proposed guidance method (without mixup) and exiting guidance method is needed. - Visualization of the covariance of the posterior would be helpful to understand the proposed method. - Additional description on how to get Eq. 8 from Eq. 7. - It is hard to see the evaluation metric as a novelty since it is not menitoned/analyzed specifically. Although there are some limitations, I will increase my rating if my questions can be reasonably answered.

Summary

This paper proposes a new score function for text driven image to image translation. The core idea is to estimate score function conditioned on both original image and the target prompt. The score function can be decomposed into two parts, one is from the target prompt and the other one is guiding term for target image generation. In addition, they also use a trick to get better masks for better preserving non-interested regions such as background.

Strengths

The idea is sound both theoretically and empirically. Their writing is clear.

Weaknesses

1, why do they choose those 5 tasks to evaluate? I know they are from prior work, but I am a bit concern that they can not demonstrate the generalizability of the method. 2, I hope they could show random samples so that we can see the average performance of their method.

Questions

na

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

na

Summary

In this paper, the authors propose a new method that can perform image-to-image translation through a pretrained text-to-image. They propose a new cross-attention map mixing technique and a new conditional score guidance function to tackle this problem. The method introduced in this paper does not require additional training. The authors have also shown strong qualitative results demonstrating various application scenarios.

Strengths

The proposed method does not require model architecture modifications or any training of the pretrained model. The qualitative results also look very promising. The paper is very coherently written.

Weaknesses

1. Equation 3 is the same as the paper “Xuan Su, Jiaming Song, Chenlin Meng, Stefano Ermon. Dual Diffusion Implicit Bridges for Image-to-Image Translation. ICLR 2023”, which the authors did not cite or compare. 2. There are a lot of approximations and conjectures in the theory which are not explained anywhere in the main paper or the appendix. 3. The presentation of the tables is a little bit misleading. It almost seems like authors choose to denote both the best and the second best results in each category just so that their CS scores won’t look too bad. 4. The authors invented a new metric to evaluate the performance. However, there is no detail of this new metric mentioned in the main paper and readers will have to refer to the appendix to understand what this metric is exactly. 5. There is no discussion of the limitations of the method, nor of any potential negative societal impact. 6. The ablation study is very limited and it doesn’t include experiments to study the effects of their newly proposed conditional score guidance.

Questions

Related to Weakness (1), can the authors compare their method to “ual Diffusion Implicit Bridges for Image-to-Image Translation”? How long does it take to generate one image?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

There is no discussion of the limitations of the method, nor of any potential negative societal impact.

Thanks for the rebuttal

Thank you for crafting the rebuttal! W1: The efficiency of the proposed method is commendable. It's encouraging to see. W2: It would be intriguing to explore the applicability of the proposed method to diffusion models without relying on an auto-encoder for latent generation. Specifically, investigating denoising within the pixel-space. I recognize that this lies beyond the scope of the current approach, so it is entirely fine. W3: The discussion regarding limitations is well-articulated. W4: Your inclusion of visualizations is greatly appreciated. A valuable addition indeed. Overall, I find that my concerns have been thoughtfully addressed. I am inclined to maintain my rating as a weak accept.

After Rebuttal

Dear Reviewer JnRy, Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any questions or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper. Best wishes, Authors

After Rebuttal

Dear Reviewer fXLA, Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any questions or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper. Best wishes, Authors

After Rebuttal

Dear Reviewer nbbt, Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any questions or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper. Best wishes, Authors

After Rebuttal

Dear Reviewer ZGMV, Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any questions or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper. Best wishes, Authors

Kindly request to respond to rebuttal from authors

Dear reviewers of #5715, Thanks for your time on reviewing this work. Since authors have provided detailed feedback regarding the proposed concerns, it would be great for you to check the response and see if your concern is addressed. Given that the deadline of discussion between authors and reviewers is approaching in a few days, your timely feedback will allow authors to further clarify. Thanks Reviewer rP2j who already responded. Yours, AC

Summary of Author Responses

We sincerely thank all reviewers for their constructive and positive comments and we present the summary of our responses to each reviewer as below. Q1. Concern about the generalizability of CSG In the main paper, our paper focused on testing CSG on local editing tasks such as cat-to-dog, dog-to-cat, wolf-to-lion, zebra-to-horse, and dog-to-dog with glasses. In addition to the local editing tasks, we employ global editing tasks, street-to-snowy street and drawing-to-realistic photo tasks, to compare CSG with Pix2Pix-Zero. As presented in Table A2 and Table A3, the proposed method outperforms Pix2Pix-Zero in terms of SD, LPIPS, and RD even with faster translation although Pix2Pix-Zero achieves slightly higher values of CS. In addition, Figure A-1 and Figure A-2 demonstrate that the proposed method also achieves better performance on the tasks. Q2. Comparison with Plug-and-Play Table 1 and Table A7 imply that the proposed method outperforms Plug-and-Play in most cases. Also, Figure D in the rebuttal document and Figure 1 imply that CSG achieves much better qualitative results. Furthermore, considering the qualitative results in Figure C and D, the proposed method even without using the cross-attention mixup outperforms Plug-and-Play. Q3. Limited editing capabilities We tried to follow the experiment protocol of prior works [Prompt-to-Prompt, Pix2Pix-Zero] by evaluating CSG on object-centric editing tasks. In addition to the object-centric editing tasks, our method can be extended to global editing tasks. To validate the property, we test CSG and Pix2Pix-Zero using global editing tasks, street-to-snowy street and drawing-to-realistic photo tasks. As presented in Table A6, CSG outperforms Pix2Pix-Zero in terms of SD, LPIPS, and RD even with faster translation although Pix2Pix-Zero achieves slightly higher values of CS. In addition to the quantitative results, Figure A-1 and A-2 demonstrate that the proposed method achieves better performance on the tasks. Q4. Incremental novelty Our algorithm is somewhat related to Pix2Pix-Zero in the sense that CSG and Pix2Pix-Zero employ the cross-attention layers in the noise prediction network for text-driven image-to-image translation tasks. However, the two methods are completely different, where CSG uses mixup strategy in cross-attention layers while Pix2Pix-Zero takes a gradient step to reduce the distance between the cross-attention maps given by the reverse process of $\mathrm{x}^{\text{src}}_t$ and $\mathrm{x}^{\text{tgt}}_t$, which requires an additional backpropagation step and leads to slower inference. Moreover, as mentioned by Reviewer fXLA, Reviewer nbbt, and Reviewer rP2j, we propose a principled technique for text-driven image translation based on the conditional score function with reasonable motivations. Such ideas have not been addressed before and are sound both theoretically and empirically, so we believe that our paper has sufficient novelty and contribution. Q5. Additional comparison with Pix2Pix-Zero Different from CSG, it is hard to apply Pix2Pix-Zero for dissimilar object-centric tasks such as hose-to-eiffel tower as presented in the right examples of Figure B since Pix2Pix-Zero enforces the shape matching through the cross-attention layers. Also, Figure A-1, A-2, and B demonstrate that CSG outperforms Pix2Pix-Zero. Q6. (Ablation) Classifier free guidance (CFG) v.s. CSG w/o mixup We test CFG using five different values of the guidance scale $s$, {3,4,5,6,7}, to compare them with CSG w/o mixup. Table A4 and Table 2imply that CSG w/o mixup always outperforms CFG with the best hyperparameter $s$ ($s$ = 3 for all tasks). Q7. (Ablation) DDIM w/ mixup v.s. DDIM To show the effectiveness of cross-attention mixup, we report the results of DDIM combined with our mixup strategy, denoted by DDIM w/ mixup, in Table A5, where the results of DDIM are presented in Table 2. The tables demonstrate that DDIM w/ mixup always outperforms DDIM except the only one case in the zebra-to-horse task. Moreover, Table 2 indicates that our mixup is also effective when combined with the proposed conditional score guidance.

Thank you for your response

Thank you for your clarification. My major concerns have been addressed in the rebuttal. I would like to change my rating from 3 to 5.