LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts

Summary

This proposed a framework for generating images from complex and detailed prompts, which took more work for previous models or frameworks. They utilize LLM's capability to generate augmented textual and visual prompts for better generations. And the framework comprises global scene generation and an iterative refinement scheme to align with conditional cues.

Strengths

- This work steps toward longer textual descriptions for image generations to ensure the fidelity of complex textual prompts. - Scene blueprints with the iterative refinement step ensure high prompt adherence recall, quantitatively validating its effectiveness.

Weaknesses

- Missing information on human study. Detailed information on human study could be provided to assess the reliability of the outcome. How do you recruit and select them based on what qualification? Isn't there any conflict of interest for the subjects and authors? How many subjects are recruited? What was the confidence of the votes? This issue is the major reason for leaning toward rejection. I am eager to see the author's feedback for reassessment. - This framework relies on recently proposed models like a strong LLM, CLIP, and an image composition model. The collection of previous works provides shallow techniques compared with them. - A missing related work. DenseDiffusion [1] may be worth being included in the layout-to-image generation subsection in Sec. 2 and the box-level multi-modal guidance in Sec. 3.4. DenseDiffusion tried to manipulate attentional weights to control the regions for layout guidance selectively. It would be appreciated if you could compare your method with it for readers in the upcoming revised manuscript. Note that, due to the narrow accessibility to this work at the time of submission, this is not considered in the score evaluation. [1] Kim, Y. et al. (2023). Dense Text-to-Image Generation with Attention Modulation. http://arxiv.org/abs/2308.12964

Questions

- Minors: - In Sec. 3.1, Models. -> Models (Please exclude the period in the section title.) - In Fig. 2, it would be inappropriate to include the logo of a commercial product (ChatGPT from OpenAI) in an academic paper. And the company may not allow the usage of their logo to promote the work.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Ethics concerns

I leave the comment that "In Fig. 2, it would be inappropriate to include the logo of a commercial product (ChatGPT from OpenAI) in an academic paper. And the company may not allow the usage of their logo to promote the work." for the possible legal issue.

Summary

This work aims to solve the problem that prior text-to-image models cannot accurately follow the object specifications in lengthy prompts. The work proposes a novel two-step pipeline that first uses a scene blueprint, which is an LLM-generated layout with object descriptions, to generate the overall image. A CLIP-based guidance is then applied to perform iterative refinement in order to make sure the content of each box is correct. The method enables accurate and diverse image generation with intricate and lengthy input text prompts. The user study and qualitative comparison indicate a non-trivial improvement over baseline methods.

Strengths

1. In addition to previous methods using the layouts for initial image generation, the method further proposes box-based refinement to improve the ability to generate all the objects mentioned in a lengthy and intricate prompt. 2. The method points out the fact that there currently lacks a pipeline for benchmarking text-to-image methods with lengthy prompts and proposes a metric called prompt adherence recall (PAR) to evaluate their method and several baselines. 3. Their method has better prompt adherence compared to baselines in prompt generation. The user study also confirms that the method can faithfully generate objects in the prompt.

Weaknesses

1. This work builds on LLM-grounded Diffusion, as mentioned in Sec 3.3. However, the difference between LLM-grounded Diffusion and the current work is not clearly explained. This work lists "scene blueprints using LLMs" as one of the contributions, but both LLM-grounded Diffusion and LayoutGPT (with both works cited in related work section) propose using LLM to generate the scene layouts with object descriptions. The authors show that the scene representation enhanced by the proposed method works better for long text prompts, but this claim of "proposing a structured scene representation" is not a contribution as it has been proposed and used in previous works. 2. The interpolation of k generated layouts could potentially cause degenerate or undesired results. For example, for "a cat and a dog, one on each side", the LLM can generate 1) a cat on the left, a dog on the right, or alternatively 2) reverse the position for the cat and the dog. The interpolation of layout 1) and 2) results in two objects placed both at the center. 3. Missing details: In Sec 3.3, the authors mention "image-to-image translation" with latent/stable diffusion that leads to generation with higher quality, but no details such as the prompts or the exact approach are given. Is this simply adding some noise and denoise? 4. The evaluation of the work does not investigate whether the proposed method degrades the fidelity of the image. The user study only involves asking whether objects are present and layouts are accurately generated rather than comparing the overall image quality. The author needs to present evaluation results that show their method does not have significant degradations on quality. If only the presence of objects is considered, a simple "copy-and-paste" from baseline text-to-image model with individual object generation using the generated layout would also result in high object presence without bringing much utility to the research community and potential applications. A typo that does not affect the rating: 1. api -> API (page 7)

Questions

The authors are encouraged to respond to and address the weaknesses in the section above.

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Summary

This paper presents a novel approach for improving text-to-image generation in diffusion-based models when processing complex scenes with multiple objects and intricate text prompts. The authors leverage Large Language Models (LLMs) to extract the layout information, detailed text descriptions, and background information from text prompts. Then the proposed layout-to-image generation model is composed of two stages: Global Scene Generation and Iterative Refinement Scheme. The Global Scene Generation phase uses object layouts and background context to create an initial scene which roughly represents the target image layout but not very accurate. Then the Iterative Refinement Scheme iteratively evaluates and refines box-level content to align them with their textual descriptions and recompose objects to ensure consistency. Extensive experiments are conducted to validate the effectiveness of the proposed approach.

Strengths

1. The proposed approach can handle the complex scenarios of text-to-image generation with long text prompts very well. 2. The paper is well-written and easy to follow. 3. The iterative refinement loop provides a possible solution to generating images of complex scenes.

Weaknesses

1. I think the major limitation of the proposed approach is efficiency. The complexity of the proposed approach increases as the number of objects increases for complex scenes. This might also be the major issue presenting this approach to be applied in real usage. 2. It seems that although the generation of objects can be iteratively refined, the bounding box locations cannot be refined. If the LLM predicts unreasonable bounding box layouts at the first stage, it cannot be corrected. Have the authors think of introducing the refinement of bounding box locations into the pipeline? 3. How many layouts are needed to interpolate? How does the number of layouts for interpolation affect the results?

Questions

Please refer to the weakness section. Post-rebuttal: I have read the author response and other reviewers' comments. I decide to keep my initial rating unchanged, but I won't fight for acceptance of this paper.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

The paper extends recent works which leverages layouts to generate scenes corresponding to complex text-prompts. This work first shows that for complex prompts, existing layout to image generation methods have certain failure modes and proposes some practical modifications which are augmented with existing layout to scene generation methods. First, the authors propose a scene blueprint to represent complex text-prompts; Secondly the authors design an iterative refinement process which improves the alignment of generated images with the complex text-prompts.

Strengths

- The research question is a very practical problem — usually most of the text-to-image generation models are not good at coherent images corresponding to complex prompts, so providing a solution for it is important. - The method consists of various components (a lot of these components are existing though) and is conceptually intuitive! - The framework obtains strong results on human-study for fidelity of images generated for long prompts.

Weaknesses

Cons / Questions - While the writing is satisfactory, it can still be improved! The authors should provide more information in the paper on how Eq. (5) is used to guide the sampling process. - Can the authors provide more intuition on the interpolation step? - Given that there are stronger open-source diffusion models (e.g., DeepFloyd) — the authors should provide some context on how long prompts work in those cases, as they use a stronger text-encoder like T5. - While the authors comment that the size of the tokens (77 in CLIP) is one of the potential reasons on why SD cannot generate compositional prompts — I believe this is only partially true. Even for non-complex compositional prompts, SD is not able to generate coherent images. Can the authors comment in general on some potential reasons why these text-to-image models are not able to generate images corresponding to simple compositional or complex prompts? I think both are related somehow and it will be beneficial to provide some context regarding it.

Questions

See Cons/Questions; Overall, the paper is practical, but the various components though intuitive are not technically novel. While I do agree that not everything in a paper needs to be novel — the authors should provide solid justifications on the design of each component. I am happy to revisit my scores after the rebuttal!

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Ethics concerns

N/A

Thank you for the valuable feedback

We sincerely thank the reviewers (XXwY, c4Qg, Br9r, kt1C) for their detailed and positive feedback. The idea of improving text-to-image diffusion models on complex prompts is a very practical research problem and is conceptually intuitive (XXwY), proposed iterative refinement approach is an effective solution for handling complex scenarios of text-to-image generation, especially with long text prompts (c4Qg and Br9r). The proposed prompt adherence Recall (PAR) addresses the lack of a pipeline for benchmarking text-to-image methods with lengthy prompts (kt1C). The proposed approach ensures high prompt adherence recall and is supported with a user study confirming the faithful generation of objects in the prompt ( Br9r, kt1C). The paper is well-written and easy to follow (c4Qg). We are pleased to observe the overall engaging comments provided by all reviewers. We have diligently addressed each concern raised by the reviewers individually, and the main PDF has been accordingly updated including a revised appendix attached at the end of paper. Below, we present a summary of our responses. - **Further Comparisons:** In response to the suggestions from **Reviewers XXwY and kt1C**, we have included qualitative comparisons with DeepFloyd and DenseDiffusion in **Figure 10** and quantitative comparisons in **Table 1** of the revised appendix. - **Guidance Function:** Addressing the recommendation from **Reviewer XXwY**, we offer additional insights into how the guidance loss is incorporated into the sampling process during the Iterative Refinement phase. - **Interpolation:** Building upon the suggestions from **Reviewers XXwY and c4Qg**, we provide further intuition about Interpolation and its impact on the final generated image. **Figures 11 and 12** in the revised appendix delve into this aspect. Additionally, responding to input from **Reviewers Br9r and c4Qg**, we present a technical explanation on handling ambiguous prompts with unclear object locations. This is supported by a study on the layout generation capability of ChatGPT, included in detail in **Table 2** of the revised appendix. - **Justification on underlying components:** In accordance with the recommendations from **Reviewers XXwY, Br9r, and kt1C**, we offer clear and comprehensive distinctions between our approach and the underlying components it utilizes. We provide justifications for each component and highlight the proposed novelties, signifying a substantial advancement in addressing a significant challenge in text-to-image diffusion models. - **Additional Details:** In response to the suggestion from **Reviewer Br9r**, we have included additional details about the image-to-image translation step used in our work. We have also included these details in **Sec. A.2** of the appendix. Furthermore, as recommended by **Reviewer kt1C**, we have provided extensive details about the user study, encompassing information about the subjects, underlying settings, and procedure. - **Efficiency:** As suggested by **Reviewer c4Qg**, compelling reasons supporting the superiority of our approach, both qualitatively **(Fig. 3 main paper and Figures 9, 10 appendix)** and quantitatively **(Table 1 in the revised appendix)**, have been included. We also outline future directions for accelerating our method. - **Study on Fidelity:** Responding to the suggestion from **Reviewer Br9r**, we have conducted another comprehensive evaluation of our approach in terms of fidelity through an extensive user study in **Fig. 13 of revised appendix**, the details of which are included in the individual response. - **Use of ChatGPT logo:** As recommended by **Reviewer kt1C**, we have removed the ChatGPT logo from the main block diagram of to avoid ethical concerns in the revised paper. **Our codes will be publicly released. All suggested changes will be reflected in our final manuscript.** We summarize the salient features of our proposed approach below: - To the best of our knowledge, our work is the first public attempt to enable diffusion models to generate images with complex and detailed textual prompts, ensuring a faithful representation of details. - We introduce Scene Blueprints as a structured scene representation encompassing scene layout and object descriptions that enable a coherent step-wise generation of images from complex and lengthy prompts. - We propose an Iterative Refinement scheme, which improves the box proposals corresponding to each object guided by a multi-modal signal in a multi-step fashion. Our refinement scheme helps in generating missing objects in the scene and provides fine-grained control over the objects. - We present quantitative and qualitative results showcasing the effectiveness of our method, in terms of adherence to textual descriptions, demonstrating its utility and benefit in text-to-image synthesis from lengthy prompts.

Response to Reviewer Br9r (2/2)

**4. Evaluation of the fidelity of the generated images** To verify the fidelity of the images generated with our method, we provide qualitative comparisons in Fig. 3 of main paper and Fig. 9 of appendix. In addition to the user study presented regarding the alignment of prompt with the generated image in the main paper, we present another extensive user study in Fig. 13 (revised appendix) only concerning the fidelity of generated images. For this user study, we compared our method with Stable Diffusion, GLIGEN, LayoutGPT and LLM Grounded Diffusion. The subjects for human study were recruited from a pool of academic students with backgrounds in computer science and engineering. To ensure an unbiased evaluation and maintain objectivity, the subjects participating in the evaluation remained anonymous with no conflicts of interest with the authors. Furthermore, we prioritized user confidentiality by refraining from collecting personal details such as names, addresses etc. The entire survey process was anonymized, ensuring that there was no leakage of data to any user. Following a 2-AFC (two-alternative forced choice) design, each participant was tasked with selecting one image out of a pair of images, having highest fidelity. Each pair of images was randomly sampled from a pool containing images from baselines as well as our approach. This system mitigates biases which can originate from user-curated surveys. Further, the pool of images was shuffled before sampling each time. The system was further equipped with the feature of randomly flipping the position of the images, to remove the user-specific position bias in case any pair of images is repeated. Additionally, a 2-second delay between questions was introduced to facilitate more thoughtful decision-making. The system yielded approximately 90 responses from 15 subjects, with each user answering an average of 6 questions. From the Fig. 13 (revised appendix), its clear that our approach compares favorably against baselines in terms of image fidelity while maintaining highest alignment with the text as measured in PAR score.