DiffUTE: Universal Text Editing Diffusion Model

Summary

This paper describes an application of diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020) to text editing. Methodologically, this work differs from previous text diffusion (Li et al., 2022) by leveraging insights on glyph encoder and OCR detector. Empirically, this work advances the state of the art for text editing by scaling these methods to larger datasets. The paper also proposes to use self-supervised training to train the diffusion model and further explore diffusion guidance.

Strengths

1. This is the work on diffusion LMs that shows results on a text editing baseline and seems to have positive results in terms of the metrics used. 2. The writing is clear and the motivations seem sound.

Weaknesses

1. The main weakness is the novelty. The core idea of this paper, i.e. latent diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on scene text editing. Most of the techniques used in the paper have been proposed perviously. 2. The author did not provide any details regarding the position control module, thus the ablation study of this part is not convincing. 3. The authors did not evaluate a variety of evaluation measures that prior work has done such as SSIM, MSE, PSNR, and many more. These metrics should be computed to a get better idea of the quality and diversity of the output. Please see these this paper for the description of these metrics: “ Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.”. 4. There are missing comparisions such as Krishnan et al 2023's TextStyleBrush[1] and Ji’s 2023’s DiffSTE [2]. 5. The model rely on a pretrained OCR encoder which just seems like an arbitrary choice. An ablation should be provided with different pretrained encoders to understand the impact of this choice. [1] Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. [2] Ji, Jiabao, et al. "Improving Diffusion Models for Scene Text Editing with Dual Encoders." arXiv preprint arXiv:2304.05568 (2023).

Questions

The author mentioned that "in the first three stages of training, we randomly crop images of sizes 𝑆/8, 𝑆/4 and 𝑆/2 and resize them to 𝑆 for training", however, which are the three stages and where is the fourth one? Moreover, an ablation that would be of interest is to train with different resolutions at different stages.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Beyond the weaknesses I listed, the authors were good at addressing several limitations of this work.

Summary

In this paper, the authors present DiffUTE, a universal self-supervised text editing diffusion model for language-guided image editing. They address the limitations of existing diffusion models by focusing on rendering accurate text and text style during image generation. DiffUTE incorporates modifications to the network structure, allowing it to handle multilingual character drawing using glyph and position information. Furthermore, a self-supervised learning framework leverages a large amount of web data to enhance the model's representation ability. The experimental results showcase the impressive performance of DiffUTE, demonstrating its ability to achieve high-fidelity and controllable editing on diverse real-world images. Overall, this paper presents a significant advancement in language-guided image editing and offers a promising approach for rendering realistic and customizable text in generated images.

Strengths

- The problem addressed in this paper is a realistic problem that current diffusion models struggle to handle effectively. - The incorporation of LLM into the inference process is a compelling and intriguing approach.

Weaknesses

- The paper claims significantly better results than other baselines in Table 1. However, it would be helpful to clarify if there are other baselines that have not been adequately considered. - A simple baseline is missing. Have the authors considered directly replacing the "source text" with the "target text" and calculating the FID (Fréchet Inception Distance)? - In Table 1, the results for SD1-FT and SD2-FT appear to be poor. It would be valuable to explain the main differences between your method and these baselines. - There is limited mention of other methods that fine-tune the encoder-decoder. How important is this step? Additionally, could you provide details on the difference in parameter numbers shown in Table 1? - The discussion regarding self-guidance is absent, despite the proposal of a self-supervised approach for achieving text editing in diffusion. The related papers are: - Self-Guided Diffusion Models - Why Are Conditional Generative Models Better Than Unconditional Ones? - Visual Chain-of-Thought Diffusion Models

Questions

As Above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The authors propose a method of fine-tuning Stable Diffusion to modify words in images, while maintaining the original font style and the background region. Specifically, they first fine-tune the VAE with text images from several datasets. Then, utilizing an off-the-shelf OCR detector, they randomly mask out one text box, and tune the denoising network asking it to fill the region with original text that is given as the condition in cross-attention layers. With this simple and intuitive method, they achieve improved performance in various evaluations.

Strengths

- the authors tackle a meaningful task - the proposed method is simple and easy to reproduce - the proposed method demonstrates improved performance on various evaluation metrics

Weaknesses

- would only work for texts that can be detected by off-the-shelf OCR detectors - there are missing details on some parts of the method

Questions

- not sure whether this method is self-supervised or not, since it requires the utilization of an OCR detector in training - could you elaborate more on a glyph image? for example, what is the output format of the glyph encoder? - could you compute the accuracy of ChatGLM’s predictions? and how much cost does it take to fine-tune it? - it seems that the figure 3 is redundant. maybe you can incorporate it into the figure 2?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

please refer to the Weaknesses section

Summary

The paper proposes DiffUTE for general text editing. DiffUTE utilizes Stable Diffusion model with several specific model designs, progressive training strategy, positional and glyph guidance, and a self-supervised training framework. Equipped with these designs, DiffUTE achieves remarkable results compared to other baselines on several public datasets. Moreover, the authors also provide a chat-based interface which enables an easier manipulation for the users.

Strengths

Originality, motivation and significance: - The paper shades an interesting perspective to edit text using pre-trained Stable Diffusion model. Two motivations raised in Line 29 and Line 32 are intuitive. - The interaction module is interesting and easy to use. Technical approach: - Finetuning VAE with a progressive training strategy (PTT) with different image sizes in different stages is a good choice to overcome blurry outputs. As shown in Table 2, with PTT, DiffUTE has a noticeable gain. - The insight into generating fine-grained texts makes sense. With positional and glyph guidance, DiffUTE generates texts with natural shapes. - Proposed self-supervised training strategy is straight-forward and useful. It also reduces the need of human annotations. Clarity: the paper offers a smooth writing and is easy to follow.

Weaknesses

- The motivation of using diffusion models v.s. GANs is not clearly stated. Why would the authors prefer to use diffusion model (e.g., Stable Diffusion)? - The paper lacks some failure case analysis. For example, DiffUTE relies on pretrained OCR detector. What if the OCR detector compromises in some cases?

Questions

Writing: - Line 129: it is better explicitly to explain what $x_m$ is at the first place.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors emit some ethical discussions in the paper. For example, the authors should discuss the misusage of the technique for misinformation spread.

Summary

This paper introduces DiffUTE, an innovative diffusion-based text editing framework designed to seamlessly fill in missing words in an image with user-specified text. By employing a self-supervised training framework, the model effectively learns from an extensive collection of synthetic data pairs, enabling it to infer accurate text styles and generate images that seamlessly incorporate the desired text. Experimental results showcase remarkable qualitative text editing performance from both the model's precision in both text and style accuracy. Additionally, quantitative analysis shows that the proposed method surpasses the performance of baseline approaches.

Strengths

+ The proposed method exhibits impressive editing performance, as demonstrated through extensive experiments. It displays a strong ability to accurately infer text styles and generate corresponding images. + Leveraging LLM, the model offers broad applicability across many possible application scenarios. + The paper is organized clearly and is easy to read.

Weaknesses

- Quantitative metrics for style: Although the paper effectively showcases the model's ability to generate text that is stylistically consistent with the rest of the images, it does not provide a quantitative analysis or specific metrics to support this claim. - Alternative diffusion-based baselines: While ControlNet is a powerful diffusion-based editing framework, it is not specifically designed for text editing tasks. Another recent diffusion-based editing approach, DiffSTE[1], shares similarities with this work as it focuses on specialized text editing and exhibits commendable performance. How does the proposed method compare to DiffSTE in terms of performance? [1] Improving Diffusion Models for Scene Text Editing with Dual Encoders.

Questions

1. Are there any metrics available for assessing the accuracy/consistency of text style? Or potentially this can be validate through a human study similar to the Cor metrics present in paper. 2. Performance comparison with other diffusion-based text-editing method, DiffSTE? 3. An intriguing aspect of this work is its remarkable ability to accurately infer text styles, even when multiple possible texts are present within an image. For instance, in Figure 4, column 1, we can observe that the imprinted time exhibits the correct style (black), despite the presence of additional red texts. It raises the question of how robust this achievement is. Specifically, does the model consistently succeed when confronted with multiple text categories? Furthermore, can this success be attributed to different types of characters, such as numerical values versus Chinese characters, as depicted in this example?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

This paper clearly discusses the limitations.