GlyphControl: Glyph Conditional Control for Visual Text Generation

Summary

This work proposes an approach to generating images with visual text. The main approach consists of a generalized version of ControlNet with rendered text as control guidance. To facilitate the training and evaluation, a benchmark dataset called LAION-OCR is proposed. Experiments show that the proposed approach outperforms competitors such as DeepFloyd IF and SD.

Strengths

- Generating visual text accurately has been a challenging issue in the field. This work provides a simple yet effective approach to this challenge by elegantly extending ControlNet with rendered text as control. I am convinced that having rendered text as input makes a lot of sense in this context and would greatly help generate accurate text. - In addition, having rendered text also allows control over the positioning, font size, which would be of great help in practice. - Experiments show that the proposed approach significantly outperforms competitors while showing compelling visual results. - Overall, I believe this work is a nice addition to the current research landscape of text-to-image generation.

Weaknesses

It would help to add some failure cases analysis in order to better understand when models may fail.

Questions

- Is the text encoder frozen? What about U-Net Encoder? What is the influence of such design choices?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations not well discussed.

Summary

This paper addresses the development of diffusion-based text-to-image generative models for generating coherent visual text. They propose GlyphControl, which augments textual prompts with glyph conditional information to encode shape details and improve accuracy. They introduce the LAION-OCR benchmark dataset and evaluate GlyphControl's effectiveness using OCR-based metrics and CLIP scores, demonstrating its superior performance over the DeepFloyd IF approach in empirical evaluations.

Strengths

The strengths of this paper lie in the proposal of GlyphControl, a glyph-conditional text-to-image generation model. Additionally, the introduction of the LAION-OCR benchmark dataset and the ability to customize and control the content, locations, and sizes of generated visual text demonstrate the paper's practical contributions to the field of visual text generation.

Weaknesses

When it comes to text generation, information such as font size and style is crucial, and it would be beneficial for the authors to conduct experimental analysis on this aspect. The authors' decision to remove images with more than 5 bounding boxes lacks clarity. Considering rich-text images as a valuable corpus and only focusing on a limited number of OCR images may limit the model's ability to generate rich-text images. While the authors utilize BLIP-2 captions, which may not consistently describe both the image content and OCR text information, it would be interesting to see how the authors address the challenge of generating captions that consider both aspects. Although this method generates accurate text, it raises curiosity about its effectiveness when dealing with a large amount of text or small font sizes, such as generating a paragraph rather than a simple phrase or word. In addition to text generation, is it possible to modify the text within an image based on given text information, such as changing the color and position of specific words? When dealing with a large amount of text, such as 500 words, ensuring text generation quality becomes crucial, as it requires high-resolution images. Additionally, ensuring efficiency in generating such images is also an important consideration. Many texts may not need to be generated as they can be obtained through text rendering. Have the authors attempted this approach, based on text rendering?

Questions

See weakness part

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The main concern of this paper revolves around the quantity of text. Starting from the database, the authors control the number of OCR boxes, which, to some extent, limits the generation of rich-text images.

Summary

The paper add glyphcontrol to diffusion model by adding controlnet that takes rendered whiteboard images as inputs. Texts in the whiteboard images are extracted by OCR engine during training and then rendered by glyph renderer. During inference, glyph renderer renders the whiteboard images based on the instructions and feed the images using glyph controlnet. They also provide LAION-OCR benchmark by filtering LAION dataset using OCR systems.

Strengths

The incorporation of whiteboard images generated by the glyph renderer into diffusion models for glyph generation, using controlnet, is a reasonable and easy to understand idea. The paper effectively demonstrates the successful synthesis of images with clean glyphs, surpassing the performance of the baselines, and the glyphs can be controlled easily. They also provide LAION-OCR dataset that is useful for OCR-related research.

Weaknesses

The proposed method requires additional training. The method may be seen as an extension of ControlNet and may appear to involve the combination of multiple modules, which could be seen as engineering work.

Questions

Does finetuning with OCR data harm the generation performance of the original model? (visual quality of non-textual parts after fine-tuning)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper does not explicitly describe their limitations. It would be nice the author provides limitations that can be solved for future works.

Summary

In this paper, a glyph-conditional text-to-image generation model named GlyphControl is proposed for visual text generation. In addition, the authors introduce a visual text generation benchmark named LAION-OCR by filtering the LAION-2B-en. The results show that method of this paper outperforms DeepFloyd IF and Stable Diffusion in terms of OCR accuracy and CLIP score.

Strengths

1. The visualization results shown in this paper are impressing. The images presented in the paper show that the text of the generated text region is accurate at both the character level and the word level. 2. The introduction of LAION-OCR dataset is a modest contribution to the realm of NLP and text image generation.

Weaknesses

1. The contribution is limited. The whole model is the same as ControlNet. The LAION-OCR dataset is just filtered out from an open-source dataset LAION-2B-en, which is not collected by the authors. 2. The results of the quantitative experiment indicate that the accuracy achieved using this method is merely 40%/26%, which to some extent suggests that the visualized outcomes have been carefully selected. More results need to be displayed. 3. The comparison is unfair. The compared methods do not prioritize text generation.

Questions

1. Does the glyph render module only support a single bounding box input or can it accommodate multiple bounding box inputs? 2. Does the method described in this paper utilize classifier-free guidance during sampling? Although caption dropping is mentioned in the experimental details, there is no specific mention of classifier-free guidance.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The manageable elements of users are still constrained. Users are unable to select the color or the font.

Summary

This work proposes GlyphControl for visual text generation by augmenting textual prompt with additional glyph conditional information. A benchmark of LAION-OCR is built for evaluating this model.

Strengths

++The task of visual text generation is interesting. ++Good results are shown in experiments. ++A new visual text generation benchmark is introduced.

Weaknesses

--The main concern is the limited technical contribution. The whole architecture of GlyphControl is a simple extension of ControlNet by using additional control of glyph images. But the idea of glyph images has been validated in GlyphDraw [13] for the same task. --I am curious by the claim of “outperforms DeepFloyd IF and Stable Diffusion in terms of 55 OCR accuracy and CLIP score while saving the number of parameters by more than 3x”. As shown in Table 1, the parameter number of GlyphControl is significantly larger than Stable Diffusion v2.0. --Why not train GlyphControl on the common SD 2.1, which has the similar number of parameters and clearly outperforms SD 2.0. --More competitive baselines should be included for comparison (GlyphDraw [13], SD XL and Midjourney). --As in GlyphDraw [13], it is necessary to report the FID values in performance comparison.

Questions

Moreo discussion on technical contribution and more comparison with competitive baselines.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

None

Looking forward to hearing the response from all Reviewers

We appreciate Reviewer Yxcw's swift response and have gained valuable insights from the constructive reviews provided by other reviewers. Your meticulous evaluations and invaluable suggestions are highly regarded, contributing significantly to the improvement of our submission. We eagerly await additional suggestions and are committed to offering necessary responses to address any concerns you may have. 🤗🤗🤗

Please read the authors' responses and start discussing.

Hi reviewers, The authors have responded to your reviews. Please carefully read their responses, consider to what extent they address your concerns, and update your review accordingly. If you have follow-up questions to anything the authors have written in their responses, I encourage you to reply to them and continue the discussion. Your AC

Looking forward to hearing the feedback from Reviewer wGfp

We extend our sincere gratitude for the invaluable guidance you have offered in the refinement of our work. Our highest priority is to rigorously address the primary concerns you have raised, particularly those related to the perceived limitations of our contributions and the fairness of comparisons. We humbly welcome any additional suggestions you may deem fit to share. Your insights hold immense importance for us, and we view them as integral to the ongoing enhancement of our work.

Thanks to All Reviewers and AC

## To All Reviewers and AC 👉 We extend our sincere appreciation for your thoughtful evaluation of our work and the invaluable insights you've shared. Your feedback has been instrumental in refining our research. As we approach the final phase of the discussion period, we wish to express our commitment to addressing your concerns comprehensively in our rebuttal. 👉 We are eager to hear the feedback from Reviewer Cb1X and Reviewer wGfp. Should you find it necessary, we are fully prepared to embark on further clarification or additional experimentation, with the aim of enhancing the robustness of our study. Your guidance is pivotal to our endeavor, and we are dedicated to incorporating your suggestions into the paper's revision process. 👉We are deeply grateful for the time and effort you have generously invested in reviewing our work. Your dedication is truly commendable, and we encourage you to reach out with any remaining questions or suggestions. Thank you once again for your invaluable contributions!