DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

Summary

This paper introduces DiagrammerGPT, a novel framework for the challenging task of diagram generation. The authors developed AI2D-Caption, a new dataset to benchmark text-to-diagram generation. DiagrammerGPT employs a two-stage process: the first stage uses LLMs to create and refine 'diagram plans' through a planner-auditor feedback loop, while the second stage utilizes DiagramGLIGEN and a text label rendering module to generate diagrams based on these plans. The framework also supports features such as open-domain generation, vector graphic creation, human-in-the-loop editing, and multimodal planning.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

(1) **Novelty and Relevance**: This paper addresses a unique and challenging problem in diagram generation, presenting a fresh perspective on a multimodal task that encompasses diverse challenges such as commonsense reasoning, layout planning, and image generation. (2) **Initial Contributions**: As a pioneering work in a new research area, this paper conducts foundational experiments demonstrating the effectiveness of the proposed method, including LLM planning with iterative feedback and the DiagramGLIGEN module. (3) **Clarity and Structure**: The manuscript is well-structured and clear, with effective use of illustrations that enhance the understanding of core concepts and methodologies, making the proposed method more accessible to readers.

Reasons to reject

(1) **Lack of Ablation Studies:** The paper does not include sufficient ablation studies to analyze the various factors influencing the performance of this complex multimodal task, making it difficult to fully understand the strengths and limitations of the proposed pipeline (further details are provided in subsequent points). (2) **Lack of Baseline Comparison:** Table 1 presents results based on constraints generated by the LLM in stage 1, but lacks a baseline comparison using ground truth diagram planning annotations. Including this baseline would clarify the T2I performance with accurate diagram plans. (3) **Limited Discussion on Diagram Planning:** The paper discusses the performance of DiagramGLIGEN in stage 2 but lacks a thorough exploration of the diagram planning in stage 1. Specifically, there is no discussion of related work in the planning stage, and only GPT-4 results are presented. It would be beneficial to evaluate other LLMs for this task and consider if state-of-the-art models in procedure planning could be applicable to diagram planning.

Questions to authors

Many questions are raised in the weakness section, please refer above. (1) In-Context Learning Details: The authors mentioned annotating 30 diagrams for in-context learning. How many diagrams were used to prompt each example? How were the supportive examples for ICL selected—randomly or using similarity functions? (2) Figure Clarity (Comment): This won't affect my rating, but some figures have text that is difficult to read due to its small size. It would be beneficial to enlarge the figures or at least the text labels for better readability.

Summary

This paper presents a text-to-diagram approach named DiagrammerGPT. It uses a two-step process to create open-domain diagrams from textual prompts, where GPT4 is prompted to generate a diagram plan followed by a visual generator creating the actual diagram from the generated diagram plan. Initially, the authors use GPT-4 to create diagram plans from text prompts during the diagram planning stage, which includes listing entities, their relationships, and layouts. Afterward, leveraging recent advances, another Language Model acts as an auditor to identify errors such as misplaced objects. The plan is then refined based on this feedback to correct inaccuracies like incorrect sizes, resulting in notably accurate LLM-generated diagram plans with minimal errors. During stage two, a new model called DiagramGLIGEN is designed for creating diagrams from plans and clearly rendering text labels for legibility. It's built upon the GLIGEN framework and trained using the AI2D-Caption dataset, which includes gated self-attention layers in the Stable Diffusion v1.4 model. To overcome the difficulty of text rendering in existing T2I models, the authors directly overlay text on diagrams to ensure clarity. Both human evaluation and LLaVA/ClIP based evaluation show that the proposed approach can generate diagrams with better quality.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. Text-to-diagram is a relatively new topic, and it is very useful in practice. 2. The proposed two-stage approach is technically reasonable.

Reasons to reject

Some important details are missing. For example, the quality of generated plan largely depends on the in-context examples. However, the authors do not introduce how to pick up a set of in-context examples in an open-domain scenario. Are the set of in-context examples selected on-the-fly or it is a static set?

Questions to authors

As I mentioned above, a lot of important details are missing. Besides the forementioned in-context examples, here are a list of points need to be further clarified: - The description of the model in Section 3.2 is somehow vague. I suggest the authors can provide a detailed model architecture diagram about GLIGEN and how the new gated self-attention layer is attached. The authors need to explain more about the intuition behind adding the gated self-attention layer, as this is an important model contribution. - AI2D-Caption is an important part of the paper. The authors state that they select 75 diagrams. What are the selection criteria? - Human evaluation for pairwise preference is mentioned at Section 4.4. Please describe how the labelling task is described to annotators to avoid potential bias, as this is important for the faithfulness of the result. - For the vector graphic diagram generation, it is not clear how this is implemented. The current version is not replicable.

Summary

The paper proposes DiagrammerGPT, a method to generate diagram images using a combination of LLMs and diffusion models. The method first generates layouts of the objects, including bounding boxes, paired text annotations, and arrows or lines between objects. This stage involves in-context learning and self-correction loops to refine the layouts. Then a GLIGEN-based layout-to-image model is fine-tuned and used for generating the diagram image based on the layouts. A new dataset named AI2D-Caption is generated based on an existing diagram dataset for the task and experiment. Experimental results show a clear improvement of the method over existing baselines.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The problem of diagram creation as image generation is interesting and has practical values. Treating it as an image generation has some advantages over program-based rendering like image diversity, flexible control, and having open-domain objects. - The proposed method, while similar to existing ones, involves arrows or lines for object relation in the first stage, which introduces more fine-grained elements to the layouts. The method also largely outperforms existing baselines, showing its effectiveness. The proposed dataset is also useful for future work.

Reasons to reject

- Technical novelty is limited. There are some new elements like the use of arrows/lines or the iterative refinement process in using LLMs to generate layouts. But most of the techniques are not new like the in-context learning design or DiagramGLIGEN (basically a simple variant of GLIGEN). - Lack of in-depth experiment/analysis. While I appreciate the significant improvement shown in Table 1&2, it is not quite obvious where the improvement comes from. Why would the fine-tuned VPGen get decreased performance in count, CIDEr, and BERTScore compared to its zero-shot version? Besides, does the improvement of DiagrammerGPT over fine-tuned VPGen come from the first or second stage? I think it would be better to compare the first stage performance of fine-tuned VPGen and the proposed method.

Questions to authors

- Does the AI2D dataset originally include arrows/lines between objects? If not, how did you get the annotation for the dataset? - Could you clarify the difference between fine-tuned GLIGEN in VPGen and the DiagrammarGLIGEN? For input, GLIGEN is like box+text while DiagrmmarGLIGEN is box+text+relation? Is GLIGEN also fine-tuned on all layers like in Table 5?

Summary

This paper presents DiagrammerGPT, which leverages large language models (LLMs) to produce text-visual diagrams. With the powerful capability of layout planning, they first derive and refine a diagram plan via LLMs. Then, they can follow the plan to generate the resulting diagram via the layout-grounded diffusion model.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

- This paper is well-written and easy to follow. - The targeted text-to-diagram is a novel task, which is a crucial application of the general text-to-image synthesis (T2I). - The proposed iterative refinement is well-motivated, and the grounded diffusion can also facilitate the output quality. - They provide comprehensive human evaluations as well as qualitative examples.

Reasons to reject

- Since there are already previous works [1,2,3] adopting LLMs to improve layout planning, the novelty of this paper can be an issue. In addition, the proposed DiagramGLIGEN is an extension of the existing GLIGEN [4]. - Although the iterative feedback seems to be helpful, there should be an empirical study to show the effectiveness of this refinement. - One powerful feature of current T2I is the diversity generation. Can LLMs also produce diverse yet meaningful diagram plans? And can those layouts further generate different visual results? Will this text-to-layout technique, which helps better controllability, hurt the diversity of T2I? - From those presented visualized cases, they look not that visually appealing for actual human usage (correct me if this is not true). I am worried about the real-world practicality with the current pipeline. **Reference** - [1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models - [2] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models - [3] VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning - [4] GLIGEN: Open-Set Grounded Text-to-Image Generation

Questions to authors

Please see Reasons To Reject

To the rebuttal

Thanks for the response! It addresses most of my questions. I remain a positive score to this paper. As a brand-new application of generative language model, I think this paper may have value to readers.

Thank you for your response

Most of my concerns are addressed in the response. I hope the authors could incorporate these results and clarifications into the next version of the paper. I have raised my score accordingly.