Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps

Summary

The paper proposes distilation for both of forward and reverse path of ODE sampling of diffusion models in order to enable faster editing.

Strengths

The overall framework is quite interesting, and applying the consistency distillation for forward and reverse process is novel. Also the method of dynamic CFG is straightforward and seems to have meaningful inversion performance improvement even with CFG. The proposed method can be combined to various image editing task and expected to show more efficient editing performance.

Weaknesses

1. Although the method shows great performance in image editing with conditions, it feels that the editing method is limited on Prompt2prompt. Is it still possible to apply the model on non-rigid editing such as MasaCTRL? 2. It seems that the most important contributions of this model is faster inversion & sampling. Please explicitly compare the inversion&sampling time with other baseline methods and put the table in main paper. 3. In the experiment part, please compare the model with other training (or fine-tuning) based methods such as instruct-pix2pix or ControlNet. The current experiment only focuses on inversion methods, therefore further experiment would be more helpful for the manuscript. 4. Does the performance degrade if the distillation step decreases? Please give more discussion and experimental results of the case of decreased steps such as single step of 4 steps. 5. minor comment, The Figure 1 (First figure) seems very redundant for me. I recommend a new figure which contains the brief summary of methodology such as combined version of Figure 3 and Figure 1. The figure in the current version make the paper seems unprofessional. As I like the overall idea, it would greatly improve the readability of the manuscript with changing the first figure.

Questions

Already written in the weakness part.

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes

Summary

This paper extend the idea of consistency distillation to inversion for image editing. By training a separate consistency model where the consistency is enforced at noise space rather than latent space. Additional cycle consistency loss is employed for more accurate inversion.

Strengths

1. The paper tackles an important problem: efficiency in image inversion. While there are lots of method improving the generation speed of diffusion models, the reverse direction of inversion is less tackled. This paper is an important contribution to this field. 2. The shows promising results in fast image inversion and editing.

Weaknesses

1. The description of the method is unclear. For example, section 3.1 and 3.2 are the main idea of the method, but it ignores an important component: the data. When training consistency distillation, we use perturbed real data, but since here we need to map to the noise, we need to have a coupling of the image and noise. How to obtain this coupling? Do we need to run inversion with the teacher model on a large dataset to obtain the training data? This is very important component of the method but I didn't find an explanation throughout the paper. 2. The method basically needs two models, by employing CD in forward and backward directions, for fast inversion and generation. Although it is practically useful, I feel like it is a bit ad-hoc. The reason is that, if we enable two models, one for fast generation and the other for fast inversion, then any diffusion acceleration methods can be employed separately for two models. It would be more interesting to study this question: given a distilled diffusion model that can be sampled in few steps, can we use this model to do fast inversion? 3. Related to above point. Two models introduce significantly more parameter overhead, and given that inversion based image editing is only only handle a small subset of image editing tasks, it becomes hard to justify whether it worths to double the parameter.

Questions

N/A

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

This work introduces invertible Consistency Distillation (iCD), which enhances text-to-image diffusion models by enabling effective encoding of real images into latent space. iCD achieves both high-quality image synthesis and accurate image inversion in just 3-4 inference steps.

Strengths

1. The adaptation of Consistency Distillation to tackle image editing tasks represents a valuable and promising research direction. 2. The design of the forward and backward consistency distillation mechanisms is interesting. 3. The experimental results are impressive, demonstrating that the proposed method achieves similar or even superior editing effects in fewer inference steps compared to existing models.

Weaknesses

1. Discussion of Failure Cases: It would be beneficial to include a detailed discussion of failure cases. Understanding the scenarios where the method does not perform optimally can provide valuable insights and guide future research directions. 2. More SDXL Demos Needed: The main paper (Figure 6) showcases demos using SD1.5, which, however, still suffers from generation issues like details of hands and faces. It is better to include more examples using SDXL (which offers improved performance) in the main paper. This would strengthen the evaluation and illustrate the capabilities of the proposed method more effectively. 3. Comparison with SDXL Turbo (Adversarial Distillation): SDXL turbo is known for its excellent performance in one-step generation and can be used for editing with tools like SDedit. A comparative analysis of the editing effectiveness and efficiency between the proposed method and adversarial distillation methods would provide a clearer understanding of the advantages and potential trade-offs, offering a more comprehensive evaluation of the proposed approach.

Questions

Please refer to the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

This work has discussed limitations in Appendix.

Summary

This paper targets a novel problem of enabling image inversion and editing for models distilled with consistency distillation. The authors identify the challenges of applying consistency distillation, which is designed for the denoising process, to the diffusion process and propose multi-boundary consistency distillation that achieves inversion with as few as 3-4 steps. A regularization term is proposed to guarantee the consistency between the forward and backward models. The authors further study dynamic CFG for performance improvements. The effectiveness is validated on SD 1.5 and SDXL models.

Strengths

- This paper targets a very emerging and important problem of how to adapt the abilities DM models have to the accelerated version such as consistency-distilled models. - The paper is generally well-organized, and the presentation is easy to follow. - The empirical results are pretty strong given the very small number of steps used by the proposed method.

Weaknesses

- While the overall problem is novel, and the solution demonstrates strong performance, almost every individual technique presented in this paper comes from another earlier work and makes the overall contribution less impactful. - The proposed framework is highly customized toward CD, and might not be able to generalize other DM model acceleration methods such as SD-turbo, DMD, and UFOGen.

Questions

The presented evaluations seem to focus primarily on 7 steps. While the performance is impressive compared to other methods, the higher performance reported by fDDIM50 makes me wonder if the proposed method can match the performance with more steps and what the marginal performance would be if we scale up the number of steps. Minor suggestion, doesn't affect rating: The overall quality of this paper can be greatly improved by replacing Figure 1. The current one seems too simple and needs to be more informative. Figure 6 can even be a better alternative for Figure 1.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have properly discussed the limitations in Appendix Section E.