Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation

Summary

The paper focuses on layout-to-image (L2I) generation from the perspective of rich-context scenario where the object descriptions are complex and lengthy. In the framework design, it introduces a novel regional cross-attention module to enhance the representation of layout regions. In the evaluation for open-vocabulary L2I models, this paper proposed two new metrics that assess model performance under rich-context descriptions, validated through a comprehensive user study.

Strengths

--The introduction of a regional cross-attention module is novel, improving the handling of complex layout descriptions compared to traditional self-attention approaches. --The paper provides rigorous experimental validation, demonstrating that the proposed regional cross-attention module enhances generation performance.

Weaknesses

--This paper presents the GFLOPs of the proposed method. However, it seems that the region reorganization and regional cross attention may affect the realtime throughput. It would be better to also analyze how this proposed module affect the runtime cost. --This paper aims to target the scenario rich-context layout-to-image generation. During training, they collect a rich-context dataset for training. Therefore, I wonder the performance gain is come from the constructed rich-context dataset or the proposed regional cross attention. --Meanwhile, the comparison with other baselines may not be fair. Since the proposed method used a different training dataset which is beneficial for rich-context layout-to-image generation. It would be more fair if the compared baselines are also retrained on the same training dataset. --This paper introduced two new metrics to evaluate the object-text alignment and the layout fidelity. However, the proposed two metrics are intuitive extensions from existing metrics, which may not be strong enough to claim as a contribution. --Regarding ablations, it would be better to have visual results to illustrate the effectiveness.

Questions

Please see above weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have discussed limitations in the supplementary.

Summary

In this work, the author views both training and evaluation of the layout-to-image synthesis task. They propose regional cross attention to addresses the issues of previous works and they also introduce a new evaluation protocols for this task.

Strengths

- I agree with the author's discussion on the desired properties for a layout-to-image model and how they achieve these properties. The contribution in terms of evaluation is also acknowledged. - Overall, the performance is superior compared to existing models.

Weaknesses

Metrics - The author view a holistic “rethinking” for both training and evaluation; however, the novelty in evaluation seems limited. For instance, using CLIP to compute patchwise similarity is not particularly novel, and although calculating SAM-driven IoU is an interesting approach and might be meaningful in open-vocab-related tasks, it does not feel exceptionally special. Method - I also agree that the one critical problem in layout-to-image tasks is overlapping issues and I am quite interested in the regional cross attention. However, despite Table 2 showing improvements with the proposed modules (including regional CA), I find the analysis of these overlapping issues lacking. Similarly, the paper argue that “desired properties for an effective rich-context layout-conditioning module” (L113) in four items, each needing more concrete analysis. Currently, the analysis seems insufficient. - The computational requirements for training seem high. How does the performance compare to other methods in terms of training costs, including both resources and data? In a current view, I am not sure that the improved performance is originated from the method itself or the increased training dataset and computational powers. Presentation - I recommend that the proposed evaluation methods be described in more detail, possibly in a more formal manner or with more detailed information such as in appendix. - Similarly, a more detailed explanation of the regional cross attention would be beneficial.

Questions

The details of models and evaluations should be included more in appendix.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The author discussed the limitation and I agree with that (as I described in weaknesses as well).

Summary

The field of paper is open-set layout-to-image (L2I) generation. They propose to apply regional cross-attention module to enrich layout-to-image generation, slightly surpassing existing self-attention-based approaches. The paper also proposes two new metrics to assess L2I performance in open-vocabulary scenarios instead of previous closed-set environments.

Strengths

1. Considering the evaluation in open scenarios is meaningful. The proposed two metrics are reasonable, and due to the use of open-source tools such as CLIP and SAM, they are relatively easy to implement. 2. In the context of rich-context layout-to-image generation, the proposed method shows a notable improvement.

Weaknesses

1. The proposed regional cross-attention may not be plug-and-play and could be incompatible with existing pre-trained large models. 2. The related work section on layout-to-image is somewhat brief. Please expand this section to provide a more detailed overview of this field. 3. In terms of quantitative metrics in COCO, which is relatively close-set, the improvement of the proposed method over the latest approaches is not particularly significant. 4. The paper lacks pseudocode for the algorithm. 5. The paper lacks the diversity analysis of the generation images.

Questions

I will consider increasing the score if the authors can address the following issues. 1. Since the authors consider the two evaluation metrics as one of the main contributions, detailed information regarding their implementation is necessary to facilitate future work. For example, specifying the model type of SAM, Clip, among other details, would be helpful. This information can be included in the main text or supplementary materials. Ideally, the evaluation code should be made open source. 2. The proposed regional cross-attention may not be plug-and-play. Please list the modifications and additional training required to apply it directly to existing pre-trained models. Providing pseudocode for the algorithm would significantly enhance the paper’s reproducibility and reduce the implementation difficulty. 3. The related work section on layout-to-image is somewhat brief. Please expand this section to provide a more detailed overview of this field. For example, Freestyle Layout-to-Image Synthesis from CVPR 2023 uses attention-based interactions to achieve open-set rich-context layout-to-image generation, which is similar to this paper. Additionally, some closed-set layout-to-image generation methods based on diffusion should also be included. For example, CVPR 2023, LayoutDiffusion applys more comprehensive evaluations than previous works, using five metrics to assess generation quality, controllability (accuracy), and diversity. As diversity of generated image is also important, it's recommended to include it.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

The paper proposes a layout-to-image generation method based on cross-attention control. It also proposes evaluation metrics for the task.

Strengths

* The paper highlights the potential effectiveness of cross-attention control and designs a learning framework using this insight. * The framework shows some empirical performance gain over prior methods.

Weaknesses

* The performance improvement compared to baselines is very limited, as shown in Table 1. What is the backbone of the baseline methods? If the backbones of these baselines are swapped to be the more powerful SDXL, they could match or surpass the proposed method. * Section 3.2 proposes four properties but there lacks a discussion on 1) why these properties are complete descriptions of the desired properties of layout-to-image generation methods, or 2) why they are necessary and if so under what scenarios or using what kind of layout or text inputs would lead to undesirable outputs. * Lines 125-127 describe a cross-attention-based baseline which could be training-free. The paper claims advantages over this baseline without empirical evidence. Quantitative and qualitative comparisons would help strengthen the claim.

Questions

* How is "open-set" and "rich-context" defined? In line 95, "open-set" seems to refer to cases when N is not fixed. But even in these cases, an off-the-shelf object detector could be used to compute evaluation metrics. Why would it be impossible to list all classes (line 181)?

Rating

5

Confidence

3

Soundness

1

Presentation

2

Contribution

1

Limitations

Limitations are discussed.

Thanks Authors for Response

I thank the authors for providing high-resolution quantitative results and explaining the performance gap in tasks with different difficulties. The explanation of cross-attention-layer training helps with clarifications. Including training details as suggested by Reviewer 2Dp7 would help further improve paper clarity. I raise my score to borderline acceptance.