Summary
The paper highlights the challenges faced by existing models in generating objects with specified counts, positions, attributes, and sizes, and emphasizes the need for compositional skills that can effectively arrange components coherently, accurately reflecting object specifications and interactions.
The authors propose LayoutGPT which aims to improve the utility of visual planning skills of Large Language Models (LLMs) by in-context learning.
The method shows promising results in generating plausible layouts in multiple domains including 2D images and 3D indoor scenes.
Experiments show that LayoutGPT outperforms text-to-image models and achieves comparable performance to human users in designing visual layouts for numerical and spatial reasonings.
In addition, LayoutGPT also shows comparable performance to supervised methods in 3D indoor scene synthesis.
The paper also proposes a new benchmark called NSR-1K for evaluating generations in terms of specified counts and spatial locations.
Strengths
1. The authors propose a novel solution for handling the issue that the current visual; generation models lack visual arrangments ability. They adopt LLMs as visual planners to generate layout information of objects in the target 2D images or 3D scenes, with the help of in-context visual demonstrations in style sheet language.
2. LayoutGPT generates reasonable layouts in multiple domains, including 2D images and 3D indoor scenes.
3. They build a challenging benchmark that characterizes counting and positional relations for text-to-image generations.
4. Experiments show the effectiveness of the proposed method.
5. The paper presents promising results and provides insights into the application of LLMs for visual planning and generation, both in terms of methodology and empirical findings.
Weaknesses
1. The writing of motivation is not clear due to a lot of inexplicable expressions.
- The claim that existing visual generative models are not equipped with various reasoning skills that exist in LLMs is dubious, given the disparate nature of reasoning skills required for visual generative models compared to LLMs. Good reasoning skills are not equal to good generative effects. Therefore, the T2I models that fail to generate objects with specific counts, positions, and attributes do not indicate lacking reasoning skills.
- In the sentence, 'But unlike LLMs, ... discrete categories ...', do the authors mean LLMs are already capable of generating layout? if yes, there should have some citations, and there is no need to further introduce the LLMs as a centralized model because there is no logical correlation with the previous text. In addition, why did the authors strengthen the discrete categories, could the LLMs generate continuous categories?
- Unclear description of the drawbacks of the existing LLMs-centered systems.
- The motivations behind constructing layouts with structured programs are not sufficiently persuasive, considering that LLMs also are trained on the plain text, and alternative methods exist for representing image layouts. In fact, the strict imposition of a structured format may pose challenges for LLMs.
- To my knowledge, visual inputs generally refer to images or videos, and the LLMs are only able to handle textual inputs. Moreover, both tasks in this work involve textual information as input, and there is no experimental evidence supporting the claim that LLMs have the potential in handling complex visual inputs.
- The meaning of "addressing the inherent multimodal reasoning skills of LLMs" requires clarification. To my understanding, LLMs already possess visual planning capabilities, and the authors' objective is to effectively leverage these skills rather than fundamentally modify or address them.
2. Inconsistent notations.
- $o_j$ vs $\mathbf{o}_j$ in the section 3.1.
- $o_j$ represents the layout of an object, so the sequence should be (c_j, x_j, y_j, w_j, h_j).
3. The utilization of CSS as a structured format for representing layouts is a bit farfetched and unnecessary. Firstly, LLMs are capable of understanding the meaning of each element in a plain sequence (o) if detailed task instruction is provided. Secondly, structured representation is not solely limited to the CSS format; alternative formats such as, 'layout: c_1(x_1, y_1, w_1, h_1), c_2(x_2, y_2, w_2, h_2), ...'.
4. The reasons behind the lower performance observed when GLIGEN generates images based on ground truth (GT) layouts during image evaluation, as well as when GLIGEN synthesizes images using layouts provided by humans, remain unexplained in Table 2.
5. It is important to note that LayoutGPT is specifically designed for generating layouts, encompassing object categories and corresponding bounding boxes. Therefore, it is unreasonable to conclude that LayoutGPT can accurately perform attribute binding. Correct attribute binding is only achieved when the attributes of objects in the generated images align with the textual descriptions. However, wrong attribute bindings between the generated images and sentences are apparent, such as the closed car window in the top row of Figure 4 and the absence of a handle on the basket in the bottom row of Figure 4.
Questions
1. The authors mention that the in-context examples are provided in reverse order based on similarity. Does the order have an impact on the effectiveness of the model？
2. How are the colored sentences in Figure 4 generated?
3. For 3D layout planning, how do the furniture frequencies in the instruction impact the performance?
4. Why does the CSS structure have more impact on task performance compared to instruction?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.