HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

Summary

This work introduces a ControlNet-based conditioning method to enable pre-trained diffusion models to be layout-conditioned. The conditioning model processes each object in parallel and fuses them as residual features for the pre-trained diffusion model. In addition to the commonly used COCO dataset for evaluation, the authors also introduce a new dataset, HiCo-7K, which is fine-grained from the GRIT-20M dataset. The generation results look good even when there are many objects, and empirical results suggest that the model is backbone-agnostic. Justification: Although the method itself does not exhibit a high degree of novelty, there are notable contributions in dataset curation. However, the dataset part is not well elaborated, and the experiments lack comparison with cutting-edge methods.

Strengths

* The results validate that the proposed method can be applied to multiple pre-trained diffusion models, demonstrating its backbone-agnostic nature. * Qualitative results are good, especially for layouts with many objects. * The authors propose a new fine-grained dataset named HiCo-7K.

Weaknesses

* The custom dataset, HiCo-7K, is an important contribution but has not been elaborated on sufficiently. The paper only provides information about the total number of images and the average number of objects. Details such as how the filtering was conducted and the criteria for manual cleaning should be included. * Table 1 does not include comparison with GLIGEN, and the SoTA method InstDiff[A], whose code was released several months before the NeurIPS submission deadline, is missing. Both of these methods can be applied on COCO and thus should be compared. * The local CLIP score in Tab 3 being higher than the Ground Truth potentially indicates that this metric may be unreliable. (This is a minor issue and my rating is not affected by this point, the author can skip this point during rebuttal if out of space) * The fuse layer is not explained in enough detail. While the sum and average cases are straightforward, the mask case requires more explanation, particularly regarding how features of overlapping objects are merged. A. Wang, Xudong, et al. "Instancediffusion: Instance-level control for image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

My major concerns are the weakness 1, 2, 4. I would like the authors to address these concerns. My final rating is subject to change based on the authors' feedback.

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

The author has discussed limitations and social impact.

Summary

This paper studies layout-to-image generation. It proposes HiCo, a diffusion model that supports a complicated, hierarchical set of bounding boxes as the layout condition. The authors also constructed the HiCo-7K benchmark to provide challenging tasks for evaluations. The experiment results show that the proposed method can effectively generate images matching the layouts in various complex scenarios.

Strengths

- The key insight of disentangling each object with one branch of HiCo-Net, a ControlNet-like conditioning branch, and fusing them into the same image, is novel, interesting, and inspiring. - The authors provided various ways to augment the HiCo-Net branches with LoRA, which makes the model more powerful and extensible. - The authors constructed their dataset HiCo-7K for evaluation. - Abundant ablation study shows the reasonability of each design choice.

Weaknesses

- The method only supports axis-aligned bounding boxes. It might be more powerful if it also supports more free-formed or precise bounding boxes like rotated squares or even polygon bounding boxes, which will make it desirable in some 3D generative tasks like layout-guided room generation. - Given that each bounding box requires an individual branch to compute intermediate features, the time complexity will be linearly growing with the number of bounding boxes. This makes the proposed method less efficient than previous method whose running time is constant. - Following the previous point, I wonder if there is a more efficient way to utilize each bounding boxes, e.g., assign a small number of nearby bounding boxes to one branch, to trade-off between efficiency and per-branch task complexity. - (Minor) In Fig.3, the "Encoder" and "Decoder" are not accurate terminologies in UNet. They should be "downsampling" and "upsampling" as in Fig.4. - (Minor) The texts in math formulas should not be italicized, which actually means the multiplication of each letter. For example, $Instruction$ means $I\times n\times s\times t\times r\times u\times c\times t\times i\times o\times n$ instead of $\mathrm{Instruction}$. - (Minor) $\times$ should be used instead of "*" in L130 and L131.

Questions

Please see "Weaknesses".

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have addressed the limitations and broader impacts.

Summary

This paper propose HiCo (Hierarchical Controllable) Diffusion Model for layout-to-image generation. HiCo Net is a multi-branch structure that is introduced to hierarchically generate the global background and foreground instances for different layout regions. The author further evaluate the performance of multi-objective controllable layout generation in natural scenes and introduce HiCo-7K benchmark.

Strengths

1.The paper writing is clear and easy to follow. 2.The HiCo model achieves spatial disentanglement and could generate more coherent and accurate images in complex scenarios. 3.The HiCo model demonstrate excellent compatibility with rapid generation plugins LORA.

Weaknesses

1.The generation requires more inference time and computational resources. 2.The bounding-box and per-object generation process could not handle more complex interactions between entities such as ‘A man on the left and his wife on the right is holding their dog in the middle.’ 3.In the context of bounding-box based layout-to-image generation, addressing the issue of overlaps between bounding boxes corresponding to different entities has been a focal point of academic discourse. However, the authors omit any discussion on how such overlaps are managed.

Questions

1.How to manage the overlapping issue mentioned above? 2.How does the runtime compare with previous works ? 3.There are similar works like LMD[1] and SLD[2], discuss the differences between HiCo and these two works. 4.Could HiCo Net be considered a variant of ControlNet within the modality of bounding box-based layout? [1]Lian, Long, et al. "Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models." [2]Wu, Tsung-Han, et al. "Self-correcting llm-controlled diffusion models."

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors are encouraged to answer the questions and address the weakness above.