Harmonizing Visual Text Comprehension and Generation

Summary

This work presents TextHarmony to simultaneously comprehending and generating visual text. The paper performs theoretical and experimental analysis about the performance degradation due to the inherent inconsistency between vision and language modalities. A MoE and LoRA-based module, Slide-LoRA, is then proposed to solve this problem by applying modal-specific and modal-independent LoRA experts dynamically. The experimental results indicate TextHarmony achieves comparable performance to modality-specific fine-tuning results.

Strengths

1. The unification of image understanding and image generation into the visual text domain is a novel approach that broadens the scope of LMM applications. 2. The results presented in this paper looks good and demonstrates the effectiveness of Slide-LoRA. 3. The paper is well organized, with reasonable motivation and insights.

Weaknesses

1. To my knowledge, there exist other multimodal generative models like DreamLLM[1] and Emu[2]. It would be better to compare with these methods. 2. The authors put related work into the supplementary material. It would be better to briefly summarize the background and representitive works in the main text. [1] Dong, Runpei, et al. "Dreamllm: Synergistic multimodal comprehension and creation." ICLR2024 [2] Sun, Quan, et al. "Emu: Generative pretraining in multimodality." ICLR2024

Questions

1. Could TextHarmony generate more complex glyphs such as Chinese characters? 2. Have you tried using OCR tools to further polish the captions of DetailedCaps-100K? For example, removing the images whose captions are inconsistent with the OCR results.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

Please refer to weaknesses.

Summary

This paper introduces a multimodal generative model (TextHarmony) for unified comprehension and generation of visual text. To overcome the performance degradation brought by modality inconsistency, the authors propose the slide-lora, which partially decouples the multimodal generation space. An image-text caption dataset, DetailedTextCaps-100K, is also developed to enhance visual text generation capabilities.

Strengths

1. TextHarmony involves visual text coprehension and generation in a single model for the first time. It achieves comparable performance to modal-specific models. It is a solid step forward for multimodal task unification in visual texts. 2. The analysis of the modality inconsistency problem in the multi-modal generation is reasonable and the proposed solution(SlideLoRA) is well motivated, novel and effective.

Weaknesses

a) The connection between this work and visual text is not clearly stated. I understand the focus of this work is to construct a multimodal generative model, but it would be helpful to elaborate specifically how this work achieves multimodal generation in the field of visual text. b) It would be helpful to report the model size and the inference speed of TextHarmony.

Questions

Line 74 ‘a versatile large multimodal’. Do you mean ‘a versatile large multimodal model’ ?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

See above

Summary

TextHarmony is a versatile multimodal generative model designed to comprehend and generate visual text. Traditional methods struggle with the inconsistency between vision and language modalities, leading to performance issues. TextHarmony overcomes this with Slide-LoRA, which combines modality-specific and modality-agnostic LoRA experts, ensuring a unified generative process. The model is enhanced by a high-quality image caption dataset, DetailedTextCaps-100K. Experiments show that TextHarmony, with only a 2% increase in parameters, matches modality-specific fine-tuning performance and improves visual text comprehension and generation tasks.

Strengths

+ Explore the possibility of integrating visual text comprehension and generation + The proposed Slide-LoRA method is effective to harmonize the training of different modalities and tasks + Experimental results show that the combination is possible and the effectiveness of the proposed model

Weaknesses

- The motivation of combing visual text comprehension and generation is not clear. Only the first try (maybe) to integrate the two tasks may not be convincing enough. - As shown in Table 1 and Table 3, the performance of the proposed method is not superior over existing baselines, e.g., TextHarmony vs. Monkey for comprehension and TextHarmony vs. AnyText for generation. It may be not easy to identify the advantages of combining visual text comprehension and generation. From this point of view, this work seems to simply try combing these two aspects, and not provides valuable research insights. - Due the limited performance improvement, the human evaluation becomes more necessary to discriminative the proposed method and existing baseline models. - More explanations or evidence are expected to support certain arguments, e.g., 1) “the optimization of TextHarmony is tremendously difficult due to inconsistent training objectives” in line 106, 2) “mutually exclusive” in line 108, and 3) why classifier and denoising problems are inconsistent as discussed in line 111? - Some unclear experimental settings, such as “w/o Slide-LoRA”, n and s in Table 4. - From the qualitative results in Figure 6, it seems that there is not any evident improvement of TextHarmony compared with AnyText.

Questions

Please refer to the above weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The interleaved generation ability of the proposed approach is unknown.

Summary

This work presents TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities.

Strengths

- This work introduces TextHarmony, a versatile large multimodal that allows for the unification of diverse visual text perception, comprehension, and generation tasks. TextHarmony performs comparably to specialized models in visual text perception, comprehension, generation, and editing - The proposed Slide-LoRA dynamically aggregates modality-specific and modality agnostic LoRA experts, partially decoupling the multimodal generative space. - A high-quality dataset of detailed visual text image captions (DetailedTextCaps-100K) is constructed with a closed-source MLLM to enhance the performance of visual text generation

Weaknesses

- Details of the Modal-Aware Gating are not given? - Results of visual text editing and generation in Table 2 demonstrate that Anytext achieves better performance. - The case in Fig. 5, "Good Time" and "Summer Love", are not correctly synthesized, and the performance is worse than Textdiffuser2 and Anytext.

Questions

see weakness

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

see weakness

Thanks for the authors' rebuttal

Thanks for carefully considering my comments and adding the human evaluation results. Some of my concerns have been addressed, e.g., the performance comparison and certain unclear arguments. Therefore, I raise my score to 5. However, compared with the existing work combining comprehension and generation, this work seems not to provide valuable insights (e.g., the synergetic relations in DreamLLM) in terms of research, even though some applications may be promoted by unifying these two.

Sincere Invitation to Participate in the Discussion

Dear Reviewer SovZ, We would like to extend our appreciation for your time and comments. Due to the rush in finalizing the writing, some aspects may cause confusion and misunderstanding. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. We are open to further discussions to clarify any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. Thank you very much for your consideration. Sincerely, The Authors

Thanks to the Reviewers

Dear Reviewers, Thanks for your valuable comments and suggestions during the review and discussion period. These feedbacks play a vital role in improving the quality of our work. As the discussion session is coming to an end, please do not hesitate to reach out to us if you have any additional comments or questions. We are more than willing to clarify your concerns. We greatly value and appreciate having this opportunity to discuss with you. Yours sincerely, The Authors