LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer

Response to Reviewer t8xx

We thank all the reviewers for their constructive suggestions, which help improve the completeness of our submission. We are encouraged that the reviews are positive in the following five levels: * The paper is "**easy to follow**" (Reviewer t8xx). * The problem we are researching is “**important**” (Reviewer Gw2u) and our idea is “**interesting**” (Review j8Mq), “**effective**” (Review j8Mq), “**practically useful**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk). * Our dataset contribution is “**useful**” (Reviewer j8Mq), “**valuable**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk, Reviewer t8xx). * Our evaluation is “**extensive**” (Reviewer Gw2u), and our results are “**SOTA**” (EPYk), “**effective**” (Reviewer j8Mq), “**excellent**” (Reviewer t8xx), and “**good**” (Reviewer Gw2u). * Our graphical system and user study are **one of the strengths** (Reviewer EPYk). We now address individual questions of **Reviewer t8xx** below. 1. **[Presentation: 2 fair?]** - This **contradicts your first strength point** that “the paper is easy to follow”, and there is no specific weakness point or question about the presentation. We therefore appeal the reviewer’s re-evaluation about the “presentation” rating and final rating. 2. **[The VAE and DETR structures are not well-motivated and validated]** - This is a **factual misunderstanding**. We have already discussed our motivations. Due to the space limit, we had to mention them in **the original submission Appendix Sec. A**. For VAE, our motivation has been articulated in **Paragraph 4**. VAE has been pre-validated by Larsen et al 2016 for image generation, and has been validated by us in **Table 3 LayoutDETR-VAE rows and LayoutDETR-VAEGAN rows**. With the help of VAE, the best results (in bold) are achieved in several metrics on several datasets. For DETR, our motivation has been articulated in **Paragraph 6 in the original submission Appendix Sec. A**. DETR has been pre-validated by Carion et al 2020 for object detection, and has been validated by CGL-GAN and ICVT for layout generation (see **Table 1**). We will manage to move this part to the main paper. We therefore appeal the reviewer’s re-evaluation about the “soundness” rating and final rating. 3. **[Technical novelty]** - We understand this is a subjective judgement, and the debate would never end. We would rather argue using Prof. Michael Black’s Guide to Reviewers: Novelty in Science (https://medium.com/@black_51980/novelty-in-science-8f1fd1a0a143). We quote some: - About the novelty of our connections between two established fields, multimodal conditioned layout generation and visual detection, Prof. Michael Black says “The novelty arose from the fact that nobody had put these ideas together before.” “Fortunately, these connections also turned out to be valuable, resulting in practical algorithms that were state of the art.” “To see the connections for the first time, before others saw them, was like breathing for the first time.” “The resulting paper embodies the translation of the idea into code, experiments, and text. In this translation, the beauty of the spark may be only dimly glimpsed. My request of reviewers is to try to imagine the darkness before the spark.” - About the novelty of our simple and reasonable reuse of LayoutGAN++ and DETR architectures, Prof. Michael Black says “I value simplicity over unnecessary complexity; the simpler the better. Taking an existing network and replacing one thing is better science than concocting a whole new network just to make it look more complex.” “If a paper has a simple idea that works better than the state of the art, then it is most likely not trivial. The authors are onto something and the field will be interested.” “The inventive novelty was to have the idea in the first place. If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea.” - About the novelty of our newly-collected dataset, Prof. Michael Black says “Novelty (and value) come in many forms in papers. A new dataset can be novel if it does something no other dataset has done, even if all the methods used to generate the dataset are well known.” - We therefore appeal the reviewer’s re-evaluation about the “contribution” rating and final rating.

Response to Reviewer EPYk

We thank all the reviewers for their constructive suggestions, which help improve the completeness of our submission. We are encouraged that the reviews are positive in the following five levels: * The paper is "**easy to follow**" (Reviewer t8xx). * The problem we are researching is “**important**” (Reviewer Gw2u) and our idea is “**interesting**” (Review j8Mq), “**effective**” (Review j8Mq), “**practically useful**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk). * Our dataset contribution is “**useful**” (Reviewer j8Mq), “**valuable**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk, Reviewer t8xx). * Our evaluation is “**extensive**” (Reviewer Gw2u), and our results are “**SOTA**” (EPYk), “**effective**” (Reviewer j8Mq), “**excellent**” (Reviewer t8xx), and “**good**” (Reviewer Gw2u). * Our graphical system and user study are **one of the strengths** (Reviewer EPYk). We now address individual questions of **Reviewer EPYk** below. 1. **[Compare to the baseline: empty space detection on a background image and the layout generation of foreground can be decoupled as two separate steps]** - In fact, we **have already compared to such a baseline and outperformed it (Table 1 in the original submission): CGL-GAN** does use saliency detection to extract the empty background first, and then learns the layout generation using GANs. We will highlight this in the next iteration. We therefore appeal the reviewer’s re-evaluation about the “soundness” rating and final rating. 2. **[The proposed dataset (images) is collected in prior work]** - This is a **factual misunderstanding**. We did collect our own images. Due to the space limit, we had to mention it in **the original submission Appendix Sec. C Paragraph 1**: In the image level, we spent non-trivial manual efforts to go through all the images in Pitt Image Ads Dataset. We then thoughtfully filtered out those with single modality, low quality, or old-fashioned designs. We finally selected qualified 3,536 images out of noisy 64,832 images in the original dataset. Moreover, we additionally searched on Google Image Search Engine with the keywords ”XXX ad banner” where ”XXX” goes through a list of 2,765 retailer brand names including the Fortune 500 brands. For each keyword search, we crawled the top 20 results and manually filtered out non-ads, single-modality, low-quality, or offensive-content images. We then selected 4,321 valid ad banner images. Combining the two sources, we in total obtained 7,857 valid ad banner images with arbitrary sizes. We will manage to move this part to the main paper. We therefore appeal the reviewer’s re-evaluation about the “contribution” rating and final rating. 3. **[The inpainted background may contain artifacts which the generator can leverage for text location prediction]** - This is a reasonable concern. In fact, we **have already considered and resolved it**. Due to the space limit, we had to discuss it in **the original submission Appendix C last paragraph**: “It is worth noting that inpainting clues may leak the layout bounding box ground truth information and shortcut training. Therefore, during training, we intentionally inpaint background images at additional random subregions that are irrelevant to their layouts.” As a result, such random training augmentations introduce additional possible inpainting artifacts that do not respond to layout locations. This avoids training from being overfitting to inpainting artifacts, and makes the inference not to focus on any such artifacts. We will move this part to the main paper. We therefore appeal the reviewer’s re-evaluation about the “soundness” rating and final rating. 4. **[The text patch has the same background patten, which may contain shortcut information for layout prediction]** - This is a **factual misunderstanding**. As mentioned in **the original submission Sec. 4 last paragraph**, the textual strings are extracted by OCR. And we condition on the textual strings rather than text patches for layout generation. There is no background pattern information for the text input and consequently no such shortcut. We therefore appeal the reviewer’s re-evaluation about the “soundness” rating and final rating.

Response to Reviewer Gw2u

We thank all the reviewers for their constructive suggestions, which help improve the completeness of our submission. We are encouraged that the reviews are positive in the following five levels: * The paper is "**easy to follow**" (Reviewer t8xx). * The problem we are researching is “**important**” (Reviewer Gw2u) and our idea is “**interesting**” (Review j8Mq), “**effective**” (Review j8Mq), “**practically useful**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk). * Our dataset contribution is “**useful**” (Reviewer j8Mq), “**valuable**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk, Reviewer t8xx). * Our evaluation is “**extensive**” (Reviewer Gw2u), and our results are “**SOTA**” (EPYk), “**effective**” (Reviewer j8Mq), “**excellent**” (Reviewer t8xx), and “**good**” (Reviewer Gw2u). * Our graphical system and user study are **one of the strengths** (Reviewer EPYk). We now address individual questions of **Reviewer Gw2u** below. 1. **[Technical novelty]** - We understand this is a subjective judgement, and the debate would never end. We would rather argue using Prof. Michael Black’s Guide to Reviewers: Novelty in Science (https://medium.com/@black_51980/novelty-in-science-8f1fd1a0a143). We quote some: - About the novelty of our connections between two established fields, multimodal conditioned layout generation and visual detection, Prof. Michael Black says “The novelty arose from the fact that nobody had put these ideas together before.” “Fortunately, these connections also turned out to be valuable, resulting in practical algorithms that were state of the art.” “To see the connections for the first time, before others saw them, was like breathing for the first time.” “The resulting paper embodies the translation of the idea into code, experiments, and text. In this translation, the beauty of the spark may be only dimly glimpsed. My request of reviewers is to try to imagine the darkness before the spark.” - About the novelty of our simple and reasonable reuse of LayoutGAN++ and DETR architectures, Prof. Michael Black says “I value simplicity over unnecessary complexity; the simpler the better. Taking an existing network and replacing one thing is better science than concocting a whole new network just to make it look more complex.” “If a paper has a simple idea that works better than the state of the art, then it is most likely not trivial. The authors are onto something and the field will be interested.” “The inventive novelty was to have the idea in the first place. If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea.” - About the novelty of our newly-collected dataset, Prof. Michael Black says “Novelty (and value) come in many forms in papers. A new dataset can be novel if it does something no other dataset has done, even if all the methods used to generate the dataset are well known.” - We therefore appeal the reviewer’s re-evaluation about the “contribution” rating and final rating. 2. **[All the quantitative metrics as well as the user study only evaluate layout quality]** - This **contradicts your strength point** that “the evaluation is extensive”. Moreover, this is a **factual misunderstanding**. We do render foreground elements on top of background images according to the real/generated layouts. See **Fig. 1,3,7 in the original submission** for the rendered samples, and **Fig. 3 caption** for the rendering procedure. As a result, all of the Image FID, Image KID, and user study are based on the photorealistic real/rendered images, so as to consider the layout-content consistency. We will highlight this in the next iteration. We therefore appeal the reviewer’s re-evaluation about the “soundness” rating and final rating.

Response to Reviewer j8Mq

We thank all the reviewers for their constructive suggestions, which help improve the completeness of our submission. We are encouraged that the reviews are positive in the following five levels: * The paper is "**easy to follow**" (Reviewer t8xx). * The problem we are researching is “**important**” (Reviewer Gw2u) and our idea is “**interesting**” (Review j8Mq), “**effective**” (Review j8Mq), “**practically useful**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk). * Our dataset contribution is “**useful**” (Reviewer j8Mq), “**valuable**” (Reviewer Gw2u), and is **one of our strengths** (Reviewer EPYk, Reviewer t8xx). * Our evaluation is “**extensive**” (Reviewer Gw2u), and our results are “**SOTA**” (EPYk), “**effective**” (Reviewer j8Mq), “**excellent**” (Reviewer t8xx), and “**good**” (Reviewer Gw2u). * Our graphical system and user study are **one of the strengths** (Reviewer EPYk). We now address individual questions of **Reviewer j8Mq** below. 1. **[Vinci also handles all these modalities]** - We have already discussed **the obvious limitations of Vinci in the original submission Sec. 2 second last paragraph**: “Vinci relies on a finite set of predefined layout candidates to choose background images from a pool of food and beverage domains. Their method is unable to design layouts conditioned on arbitrary backgrounds in open domains, natural or handcrafted, plain or cluttered, like ours.” We agree with the reviewer to lower our key in the next iteration. 2. **[Computation cost and runtime analysis]** - In fact, we have already measured training and runtime computation costs. Due to the space limit, we had to report them in **the original submission Appendix Sec. B 2nd last paragraph**: “We train on 8 NVIDIA A100 GPUs for 110k iterations in 4 days. Inference is much more efficient: we load only G into a single NVIDIA A100 GPU and it consumes only 2.82GB of memory. It takes only 0.38 sec to generate a layout given foreground and background conditions.” We will manage to move this part to the main paper. - Following the reviewer's suggestion, we also measure and summarize the space complexity and training/runtime time complexity of our model variants and all the baselines below. Although our training is relatively costly, it is just one-time. Our runway cost is one of the most efficient. | Method | #parameters (M) | Training on 8x A100 GPUs (hours) | Runtime on 1x A100 GPU (seconds per layout) | |-------------------|:---------------:|:-----------------------:|:-----------------------------------:| | LayoutGAN++ | 252 | 52 | 0.38 | | READ | 310 | 89 | 0.38 | | Vinci | 251 | 51 | 0.36 | | LayoutTransformer | 190 | 36 | 0.39 | | CGL-GAN | 88 | 77 | 0.40 | | ICVT | 61 | 34 | 0.44 | | LayoutDETR-GAN | 270 | 96 | 0.38 | | LayoutDETR-VAE | 189 | 35 | 0.38 | | LayoutDETR-VAEGAN | 351 | 99 | 0.38 | 3. **[Show diversity of the generation]** - In fact, we have already discussed the diversity of generation. Due to the space limit, we had to mention it in **the original submission Appendix Sec. E Paragraph 2 and Fig. 8**. 4. **[The limitation of the proposed method]** - In fact, we have already discussed the limitation. Due to the space limit, we had to mention it in **the original submission Appendix Sec. F and Fig. 9**. 5. **[How to distinguish foreground and background images?]** - For our ad banner dataset and CGL dataset, we intend not to distinguish foreground and background images as there is no foreground bounding box ground truth collected. All the visual contents except texts are regarded as the background image condition. On the other hand, because CLAY dataset has the ground truth annotation for all the foreground visual elements, all the training and testing on CLAY are conditioned on textual plus visual foreground inputs (e.g. icons). 6. **[Why is Image FID pretrained on ImageNet a useful metric?]** - Image FID and Image KID are standard metrics to evaluate the photorealism between real and generated images. In our case, the real/generated images are rendered by overlaying foreground elements on top of background according to the real/generated layouts. Considering foreground and background images contain natural images, Image FID and Image KID are meaningful to measure the layout-content consistency.