Summary
This work points out that existing diffusion inversion methods achieve good reconstruction but limited editability due to their separate treatment of inversion and editing processes. To address this limitation, the authors propose a new diffusion inversion method, TODInv, for high-fidelity text-based real image editing.
The authors first categorize text-based image editing prompts into three classes: structure editing, appearance editing, and global editing. Inspired by the extended prompt embedding space P*, they observe that different layers in the U-Net architecture exhibit varying levels of importance depending on the editing class. Consequently, they perform diffusion inversion through prompt optimization in P* space only on the relevant layers for each editing class. They demonstrate the effectiveness of TODInv through extensive quantitative and qualitative experiments.
Strengths
1. The paper is well-motivated: Even within text-based image editing tasks, different editing prompts require distinct editing capabilities, making it challenging to faithfully execute all editing prompts with a single unified approach. This paper effectively identifies this limitation and proposes a new method based on this observation.
2. The method is simple and the paper is easy to follow.
3. The paper provides extensive quantitative and qualitative experimental results across various editing methods and inversion methods.
Weaknesses
1. Poor presentation
* The paper contains several errors in highlighting the best values in bold within tables. Most of these errors occur when the proposed method's values are incorrectly marked in bold despite not being the best performing results. These errors could potentially mislead readers into overestimating the performance of the proposed method and significantly diminish the credibility of the reported experimental results.
* There is a significant formatting error in [page 7, line 343] where text lines overlap. While such formatting issues might seem minor, given that the submission has reached the 10-page limit, resolving this overlap would likely cause the paper to exceed the page limit - a violation of ICLR's submission guidelines.
2. Limitations of proposed methods
* The paper's core concept of "task-oriented" begins with classifying given editing prompts, but this classification relies on ChatGPT. However, the paper lacks quantitative analysis regarding ChatGPT's classification performance. Moreover, this dependency should be accounted for in the inference time measurements.
* The method's requirement to pre-define class-specific editing layers within the U-Net architecture introduces architectural constraints that limit its broader generalizability.
3. The lack of implementation code limits reproducibility and credibility of the reported experimental results.
Questions
1. Why does the method exhibit inferior performance at T=10 and T=100 compared to T=50? This observation appears counterintuitive, as diffusion models typically demonstrate improved performance with an increasing number of timesteps.
2. To substantiate the claim that ChatGPT can perform editing task classification, a quantitative analysis demonstrating its classification accuracy on the PIE-Bench dataset images using the ChatGPT API is necessary.
* Given the high operational costs associated with the ChatGPT API, exploring the use of open-source MLLMs presents a viable alternative. Additionally, fine-tuning a smaller model on task-specific datasets for editing classification could potentially offer a more cost-effective solution while maintaining performance.
3. Can we use TODInv on DiT [1] style models too? Recent emergence of DiT architectures as alternatives to UNet raises questions about TODInv's potential applicability to DiT models.
4. Why do the structural similarity results exhibit notably strong performance for P* while demonstrating significantly weaker performances for P, Pt, and P+ in Table 2?
[1] Scalable Diffusion Models with Transformers