TextDiffuser: Diffusion Models as Text Painters

Summary

This paper proposed TextDiffuser to achieve accurate text rendering for diffusion models. TextDiffuser generates character-level text layouts to guide the text rendering process (image generation). The model is evaluated on 3 tasks including text-to-image, text-to-image with template, and text inpainting to demonstrate its flexibility and controllability. Additionally, the paper contributes the first large-scale text image dataset with OCR annotations, named MARIO-10M.

Strengths

1. The paper is well motivated, which tries to solve the problem of generating text image with diffusion model. Also, the paper is well organized and easy to follow. 2. The TextDiffuser model is flexible, which can be adaptive with different conditions as shown in the experiments. 3. A new dataset specifically designed for text rendering is proposed.

Weaknesses

1. Mismatch of character level layout and the real text. The character level layout is generated from standard fonts (rendered using toolkit such as opencv), however, the real character can be of any style or font, which leads to a mismatch of character level layout. In another word, guided by such a layout, the generated text will tend to have the same character interval and aspect ratio of the standard font, which means it will limit the generation diversity of fonts. 2. Generation diversity has not been considered. There exist several reasonable text styles for the same background image. As the proposed model cannot explicit control the attribute of texts such as color, layout, font, style, etc., it is important to measure the generation diversity of the model. 3. The performance comparison seems to be limited to general text-to-image diffusion models with no specific optimization on text painting. A comparison with some state-of-the-art text generation methods is necessary to show the quality of text generation.

Questions

1. How does the model perform on more complex characters such as Chinese characters? A simple layout guidance seems to only constrain the position and order of letters, without any specific improvement on the quality of generation details. I wonder if there is any method to improve the adaptiveness on other language?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations and potential impacts have been discussed.

Summary

The paper proposes TextDiffusers, a 2-stage model to generate images with text per the input text prompt. TextDiffusers also supports text inpainting. The two stages consist of (1) estimation of layout of keywords (inspired by Layout Transformer) to get character-level segmentation masks and (2) image generation conditioned on the character-level segmentation masks. The paper also contributes to MARIO-10M, a large data set of image-text pairs along with rich OCR annotations. Benchmark MARIO-Eval with 5414 prompts is created from MARIO-10M and few other existing datasets, in order to evaluate quality of rendering text in the image. Stage 1 consists of novelty such as encoding of width of keywords. Encoding width of keywords improves IoU, especially for shallower Layout Transformer. Stage 2 introduces character-aware loss and extends the denoising loss to include character level segmentation masks. The paper contains implementation details for reproducibility, comparison against the states of the art demonstrating large improvements in OCR quality metrics, comprehensive ablations studies, several qualitative samples and limitations.

Strengths

S1) Novel approach to generate images with text S2) New data set for MARIO-10M along with benchmark MARIO-Eval S3) Comprehensive experiments demonstrating large improvements against various states of the art on OCR metrics

Weaknesses

W1) While the methodology introduced by the paper seems solid, the main weakness is comparison against strong baseline which was also trained on training split from MARIO-10M. Appendix J alludes to equivalency of fine-tuning of pre-trained model, but Stable Diffusion model is not the best when it comes to producing visual text. It is understandable that official code for Imagen, Parti are not available. Perhaps DeepFloyd could have been fine-tuned on MARIO-10M. From Table 4, DeepFloyd is also better on Fidelity (lower FID score). It is not clear if it would perform better than TextDiffusers with this additional fine-tuning. W2) Information and comparison of model size and training/inference latency are missing. W3) Appendix N / Figure 14 in the supplementary material includes a handful of sample generations without text. However, it is not clear if TextDiffusers loses its capability to generate such images in the general sense. Qualitative and Quantitative evaluations to test this are missing.

Questions

Please see list of weakness. Q1) According to Line 188, the maximum length of tokens (L) is limited to 77. Do all captions in the dataset contain <= 77 tokens? Q2) Related to Q1, DeepFloyd uses T5-XXL which is much more powerful (as observed by Imagen). Although L<=77, do you have insights on the richness of embedding that may give it a higher edge over TextDiffusers?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

Yes

Summary

This work presents an approach to enhance the text rendering ability of a text-to-image diffusion model. The authors introduce a new diffusion model called text-diffuser that leverages image captions and text segmentation masks to generate text images. They also collect a large-scale image dataset MARIO-10M, which includes explicit text information such as segmentation masks and OCR annotations. The experimental results indicate a significant improvement in text generation ability but a slight decrease in image quality.

Strengths

* The large-scale text image dataset with text annotations MARIO-10M is commendable is valuable for further research in this domain. * Incorporating character-level segmentation masks to enhance the text rendering ability of a diffusion model is a novel idea. The results presented in the paper demonstrate the effectiveness.

Weaknesses

* The paper mentions that the layout generation model is trained to generate text segmentation masks, which might not be suitable for scene texts with significant perspective changes. Most of the generation examples lack realistic text styles with perspective changes and complicated text layouts. * Some GAN-based works [1, 2, 3] exploring scene text editing and a recent diffusion-based scene text editing work [4] could be relevant for part-image generation evaluation. Authors could consider incorporating a discussion and comparison with these related works. [1] Wu, et al . “Editing Text in the Wild.” https://doi.org/10.1145/3343031.3350929. [2] Qu, et al. “Exploring Stroke-Level Modifications for Scene Text Editing,” https://ojs.aaai.org/index.php/AAAI/article/view/25305 [3] Krishnan, et al. “TextStyleBrush: Transfer of Text Aesthetics from a Single Example.” http://arxiv.org/abs/2106.08385. [4] Ji, et al. “Improving Diffusion Models for Scene Text Editing with Dual Encoders.” http://arxiv.org/abs/2304.05568.

Questions

* It would be interesting if we can control the style of generated texts through language descriptions on textdiffuser. For example, the image s generated with this caption: a bear holding a board with a **red/purple/yellow** "hello world"? It would be beneficial if the authors could provide some generation examples showcasing this capability.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Authors discussed the limitation of the trained model, including small text generation and multiple-word generation.

Summary

This paper introduces a model for generating visually appealing and coherent text within diffusion models. It also presents the MARIO-10M dataset and the MARIO-Eval benchmark for evaluating text rendering quality. Experimental results demonstrate their method's flexibility and controllability in creating high-quality text images and performing text inpainting.

Strengths

The strengths of this paper include the proposal of TextDiffuser, a flexible and controllable framework based on diffusion models, with two stages for layout generation and fine-tuning.

Weaknesses

The paper does not explicitly mention how the model handles the generation of rich-text images when the number of queries is limited. However, it raises concerns about the efficiency of the model when dealing with a large number of queries. When there is a sequential relationship between boxes in the context of generating rich-text images, the paper does not specifically address how the model handles the ordering of boxes. While part-image generation seems reasonable, it could be computationally slow when dealing with a large number of texts as it requires predicting numerous positions. The paper does not mention the specific number of texts per individual image in the mentioned databases in the main paper, which could be important information to provide. Regarding image resolution, the paper suggests that it can be low when dealing with single texts. However, when generating a large number of texts, high-resolution generation becomes necessary. It is essential for the authors to address how they ensure the accuracy of each generated text and the efficiency of text generation in such cases. Using sequential generation for layout masks is an interesting point. Have the authors considered directly using diffusion to predict layout masks? It would be beneficial to have more comparative analysis between the two methods.

Questions

See the weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The rationale behind this paper, using seq-2-seq generation, is reasonable. However, there is not a clear explanation of how the generation of rich-text is achieved.