LayoutPrompter: Awaken the Design Ability of Large Language Models

Summary

This paper presents a set of few-shot techniques to enable large language models to generate geometric layouts, particularly in the domain of User interfaces, posters and documents. These techniques include dynamic prompting and layout ranker modules, which allows the LLMs to learn from relevant examples in the dataset and filter out badly generated UIs from various samples. The presented technique enables cross-domain, task-generic generation of layouts without the need for training, utilizing existing knowledge in LLMs which was unexplored in prior work.

Strengths

- Enabled LLMs to perform a novel task. This task is also typically challenging for LLMs to perform due to its highly geometric nature. - Experiments and prompts that lead to the success of layout generation in LLMs would constitute a significant contribution, given the challenging nature of this task in LLM. - Introduced dynamic prompting and layout ranker, of which their formulation and implementation can be useful for future layout generation work - Achieved competitive performance in some layout generation tasks, without the need for training

Weaknesses

Some of these weaknesses are pointed out in the limitations sections by the authors themselves, but I would like to highlight them since I believe they are important for the significance of the current paper: - **Reliant on existing dataset's coverage of the target task to achieve good performance:** This is shown in the ablation study that the FID is significantly higher without dynamic prompting. This is an important issue in the UI domain where data is difficult and costly to collect. - **Generalisation ability of LLMs in generating UIs:** Since the dynamic prompting module relies heavily on the provided prompt, it is unclear whether the generated UIs are novel and are meaningful variations for design inspiration. This can be shown by computing the similarity between the generated UIs and the prompt UIs, and comparing this with existing work (which might be generated UIs against the entire dataset for existing techniques that don't use prompts). - **Diversity and Coverage of the generated UIs:** Related to the above point, it is unclear how much of the entire design space that the generated UIs can cover. Existing work computes the Wasserstein distance between that of the generated UIs and the test set, of which this work did not compute (partially due to the conditional nature of all evaluated tasks). I wonder if the author can perform some unconditional UI generation experiments to explore the knowledge of the overall UI design space that is learned by existing LLMs.

Questions

These questions are directly relevant to the weaknesses of the paper: 1) What is the average qualitative and quantitative similarity of the generated UIs against the most similar UIs in the prompt? It would be great if the author(s) can use an established metric (e.g., DocSim) and compare this similarity against existing work. Existing work that doesn't use prompt can be compared against the most similar entry in the entire dataset. 2) What is the pool of UIs that the prompt UIs can be sourced from? Is this only from the training set of various domains? 3) How does the performance of the model correlate with the size of the dataset that the prompt UIs can be sourced from? It would be great if the author(s) can perform some ablation experiments along this dimension. 4) Can the author(s) discuss further the significant drop of FIDs w/o dynamic prompting? This discussion could be qualitative.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are adequately addressed.

Summary

The authors propose LayoutPrompter, which is a new approach for generating high-quality, cross-domain graphic layouts without any model training or fine-tuning. It can handle a variety of conditional layout generation tasks and outperforms existing methods on some metrics. The approach is competitive with state-of-the-art approaches on five traditional conditional layout generation tasks and even extends to solve two challenging problems that have more flexible constraints.

Strengths

The strength of this submission lies in its clear motivation, well-written content, and ease of understanding. The authors have effectively articulated the purpose and significance of their research, making the motivation behind the study readily apparent to the reader. The paper is written in a coherent and structured manner, which contributes to its ease of follow throughout. Furthermore, the study impressively conducts solid experiments that encompass a range of layout design settings, providing robust empirical evidence to support their findings. Overall, the submission showcases a strong foundation and execution, making it a valuable contribution to the field.

Weaknesses

(1) One major weakness of the submission is the lack of detailed information regarding the new dataset, WebUI. While the authors mention its collection, they provide very limited information about the data collection process or the format of the dataset. This omission raises several questions that remain unanswered. It would be beneficial for the authors to include a section in the main paper or supplementary material that thoroughly describes the data collection process, including the sources, selection criteria, and any preprocessing steps applied. Additionally, specifying the format of the dataset (e.g., file type, structure) would provide valuable insights for readers. (2) Another weakness pertains to the organization and clarity of the dataset setup explanation. As the submission covers multiple tasks and settings, it would greatly enhance readability if the authors were more specific about which dataset was used for each setting. Currently, the explanation of the dataset setup (lines 227-236) appears disorganized and challenging to follow. (I searched for the experimental results on WebUI and found nothing, then it took me a while to figure out that WebUI was referred to as text-to-layout…) To address this, I recommend the authors consider incorporating a table that lists the tasks and corresponding datasets. This table would help readers quickly identify the datasets utilized for each specific setting, reducing confusion and facilitating a more straightforward interpretation of the experimental results.

Questions

(1) Figure 1 is a bit hard to understand as a teaser figure. It took me a while to figure out that the left parts refer to the “ five typical tasks and two more challenging tasks”, while the right parts are the subdomains of discussion… (2) Missing reference for saliency map on line 161 (3) Are there any ablation studies on the effect of the number of prompting examples? (4) Regarding the new dataset: - What is the source of the website? - What kind of HTML elements are included? - What is the portion of each element? - How was the quality controlled (e.g., removing ads, removing unrelated banner, etc)? - How was the instructions for layout design collected? - Does the instruction creation process involve human effort or was it automatically generated? - Where is the data sheet for this new dataset? (5) Out of curiosity, may I ask why the authors did not conduct any human evaluation?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have included discussions of the limitations.

Summary

In this paper, the authors introduce LayoutPrompter, a novel approach for conditional graphic layout generation using large language models with minimal demonstrations. The authors frame the graphic layout generation task as a sequence generation problem. They represent layouts as HTML files, leveraging the fact that large language models are pretrained on a corpus that frequently includes HTML. A unique aspect of this approach is the use of a Dynamic Prompting strategy. In this strategy, the model retrieves exemplars that are more closely related to each test sample based on a defined constraint similarity metric. This ensures that the in-context examples used for prompting are highly relevant. Furthermore, a Layout Ranker is appended to the model, which selects the optimal layout post-inference. Notably, the method does not require extensive training data and demonstrates adaptability across various layout domains. To validate the efficacy of LayoutPrompter, the authors carry out experiments across different layout domains. The performance of LayoutPrompter is benchmarked against various task-specific and task-agnostic baselines. Remarkably, LayoutPrompter yields competitive results even without domain-specific training or fine-tuning. It is worth noting that LayoutPrompter is particularly effective on a custom WebUI dataset, which is constructed to investigate a new text-to-layout generation task. This outperformance is more pronounced when compared to results on other layout domains and against other baselines.

Strengths

Originality: LayoutPrompter is innovative in using large language models for layout generation without fine-tuning. The adaptation of dynamically choosing better demos (Dynamic Prompting) and output verification (Layout Ranker) to this field is a clever utilization of existing techniques. Quality: The method's solid integration of Sequence Generation, HTML representation, Dynamic Prompting, and Layout Ranker underpins its robustness. Comprehensive benchmarks add credibility to the research. Clarity: The paper is well-structured, with clear explanations of methodology and results, making it accessible and easy to follow. Significance: LayoutPrompter's ability to generalize without extensive training data is noteworthy, indicating the potential for diverse applications and research in layout generation.

Weaknesses

Limited Novelty in Techniques: The application of large language models for layout generation is noteworthy, but the utilization of HTML Representation, Dynamic Prompting, and Layout Ranker lack innovation. It would be interesting to see the exploration or introduction of new methodologies tailored for this domain. Inconclusive Ablation Study: The ablation study on the Dynamic Prompting and Layout Ranker does not convincingly demonstrate the effectiveness of these two techniques. Insufficient Experimental Analysis: While the paper presents experimental results, it lacks in-depth explanations and analyses. Elaborating on the implications of these results, and providing a more detailed interpretation, would add substantial value. In-depth Analysis of Failures: The paper could provide a more detailed analysis of cases where LayoutPrompter fails or underperforms. Understanding the limitations and areas where the model struggles would offer insights for future research and improvements. Writing Clarity and Organization: The paper could improve in terms of clarity and organization. For instance, in discussing the motivation for bridging the gap between existing methods and human designers, the text lacks coherence and the last two points seem to overlap. Clearer and more structured presentation of concepts would enhance the paper’s readability.

Questions

Ablation Study Details: Could the authors elaborate on the methodology used for the ablation study, and provide insights into why Dynamic Prompting and Layout Ranker might not have shown significant effectiveness. Would considering different variations of these techniques lead to different outcomes? Detailed Experimental Analysis: Could the authors provide a more detailed analysis of the experimental results? Specifically, why LayoutPrompter performs better on certain metrics compared to others. Moreover, what are the factors that contribute to its varied performance across different domains? Generalization Across Domains: The paper claims that the method can be generalized across domains. Can the authors provide additional empirical evidence or examples to substantiate this claim? Specifically, how will the model perform in a zero-shot scheme or given demos in different domains? Handling of Failures and Underperformance: It would be helpful if the authors could elaborate on the specific cases where LayoutPrompter fails or underperforms. What are the common characteristics of these cases and are there any strategies that could be employed to mitigate these issues? Clarity and Organization of Writing: There are instances where the writing is not very clear or well-organized, as observed in the explanation of closing the gap between existing methods and human designers. Could the authors clarify the intended message and possibly consider restructuring this section for better clarity?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper leverages large language models to solve the layout generation tasks. It shows a great performance when generalizing to other domains or other tasks. Since in-context learning is used, there is no need to train the model on a vast amount of data.

Strengths

1. The prompting method shows a good performance and does not need large amount of training data. 2. The new method can well generalize to new domains and new tasks without an additional model

Weaknesses

1. The paper used text-davinci-003 as LLM, but it is also worthwhile to see how other LLMs perform with the proposed prompt-based method. Can these prompts perform well on other LLMs, including the in-house or open-sourced ones. 2. The in-context learning is not stable with LLMs. The sample prompt may have different results depending on the random seed and temperature parameters. The paper didn't report the variance of the results. 3. In line 258, it is also necessary to mention the parameters for running LLM to repeat the experiments.

Questions

See weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The authors propose a new conditional layout generation method using LLMs with few-shot learning. The method uses a pre-trained GPT-3 with two techniques: dynamic prompting and layout ranker. Dynamic prompting is to provide few-shot examples with similar constraints to the input query using ConsSim, and layout ranker is to rank the layout using weighted metrics. The authors show that this method can generate layouts that comply with different types of conditions across domains, and the qualitative and quantitative results show that the method is comparable and sometimes better than existing training-based, domain-specific methods on various metrics.

Strengths

1. The proposed method addressed the problems of existing methods: reducing the training cost using few-shot learning and providing abilities across domains and tasks by properly designed input/output formats and pre-trained LLMs. 2. In addition to generic layout conditions, the proposed method can be applied to two more challenging tasks: content-aware generation and text-to-layout. 3. The two techniques, dynamic prompting and layout ranker, are simple and effective without extra fine-tuning or complex heuristics. 4. The authors demonstrate the possibility of leveraging LLMs for design-related tasks.

Weaknesses

1. It will be great to see more comparisons and discussions on content-aware layout generation. For example, some prior works, such as CanvasVAE [1] and CGL-GAN [2], are missing here. 2. It would be great if the authors could examine if the retrieved examples are similar to the given test set layout. For instance, in RICO, multiple layouts might be extracted from the same app, so they might look very similar and provide a way for the model to cheat. 3. Ablation studies can be improved by adding, for example, zero-shot results, results of different ways to represent the constraints, results of varying screen resolutions, results in groups with different numbers of elements, etc. 4. It would be great if the authors could conduct a user study, ideally on designers, since none of these metrics can perfectly capture human perceptions/judgments. [1] CanvasVAE: Learning to Generate Vector Graphic Documents, Kota Yamaguchi, ICCV 2021 [2] Composition-aware Graphic Layout GAN for Visual-textual Presentation Designs, Zhou et al., IJCAI 2022

Questions

1. The Gen-T and Gen-TS results in RICO in Fig 5 look very similar, and I wondered if the authors intentionally chose them for comparison. Otherwise, it seems impossible since the conditions are generic and have many possible solutions. 2. In the ablation studies (Table 5), the Gen-T results w/o HTML achieve 0% constraint violations, and the proposed method performs the worst. Are there possible reasons? The other tasks do not have results w/o HTML, and I wondered if the same situation can be observed. 3. It would be interesting to see the results in the refinement task with different perturbation levels. 4. Have the authors tried fine-tuning LLMs on these tasks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes, the authors have addressed the limitations.