Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation

Summary

This paper employs scene graph for image generation. Different from the previous methods, they employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. The authors propose a semantics-Layout Variational AutoEncoder to jointly derive layouts and semantics from scene graph. Then they develop CMA integrated with a diffusion model. They also introduce the multi-layered sampler for achieving graph manipulation. Experiments show that the method outperforms existing methods.

Strengths

1. The paper address the problems in existing methods well. Existing methods in the field of scene graph to image generation mainly depends on the layout or semantics. Using one of them may cause some problems. Inspired by these phenomenons, the authors propose the method to jointly considering the layout and the semantics. What's more, the techniques used in the framework are novel enough. 2. The authors conduct plenty of experiments. The ablation studies support the motivations.

Weaknesses

The only weakness I found is that the authors should reorganize the paper carefully. The writings is not so clear in some sections. For example, the multi-layered sampler section is too abstract to be understood.

Questions

No.

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper proposes DisCo (Disentangled Compositional image generation), which integrates both layout and semantic information derived from scene graphs to improve the quality and controllability of generated images. In particular, DisCo has three main components: Semantics-Layout Variational AutoEncoder (SL-VAE) for disentangling spatial layouts and semantics, Compositional Masked Attention (CMA) for fine-grained attribute guidance, and Multi-Layered Sampler (MLS) for object-level graph manipulation. Extensive experiments demonstrate that DisCo outperforms state-of-the-art methods in generating rational and controllable images from text, layout, and scene graph conditions.

Strengths

1. The motivation is clear. The idea of disentangling layout and semantics from scene graphs is novel. 2. DisCo outperforms recent methods in both fidelity and diversity of image generation, as evidenced by the IS and FID scores. Overall, it enhances the generation diversity and controllability. 3. Extensive experiments and ablation studies have demonstrated the effectiveness and the contribution of each component.

Weaknesses

1. The increased inference cost of DisCo (Table 7). In particular, CMA mechanism might increase the computational cost, which may limit the method's scalability and efficiency, especially for large-scale applications. Moreover, since diffusion models are already quite large, the additional AutoEncoders (Lines 129-130) may result in more parameter and memory overhead. 2. DisCo requires expensive training, e.g. 4 A100 GPUs, as indicated in Lines 202-203. With more models releasing recently, this technique might be not scalable. 3. The image quality looks better with this proposed method. However, as metrics today cannot always reflect the real image quality, it would be more convincing to conduct a user study, e.g. votes, to quantify the advantage of DisCo compared to previous works.

Questions

See the weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, authors have stated their limitations on attribute leakage of overlapping in Section A.6, as well as a short discussion on the broader impact in Section A.7. Beside that, authors are also encouraged to add a few discussions on the efficiency of their proposed pipeline. Even though these overheads are inevitable (they are quite common in most researches), a clearer trade-off between the improved image quality and the increased model complexity would help to better assess the value of this work.

Summary

This paper presents "DisCo," a novel framework for generating complex images from structured scene graphs. Unlike traditional text-to-image or layout-to-image methods, DisCo utilizes a Semantics-Layout Variational AutoEncoder (SL-VAE) to disentangle and generate diverse spatial layouts and interactive semantics from scene graphs. It incorporates these elements using a Compositional Masked Attention (CMA) mechanism within a diffusion model, enhancing generation control and rationality. The framework also introduces a Multi-Layered Sampler (MLS) for flexible, graph-based image editing, preserving visual consistency while manipulating object attributes and positions.

Strengths

1. Introduces innovative methods for disentangling and integrating spatial and semantic information from scene graphs, which is a novel approach in image generation. 2. Offers significant improvements in image generation from complex scene graphs, enhancing both the fidelity and controllability of generated image

Weaknesses

1. The paper lacks quantitative comparisons with closely related baselines, such as R3CD, which could provide a more comprehensive evaluation of the model's performance. Inclusion of these comparisons could help validate the proposed advantages of DisCo over existing methods, particularly in handling complex scene graph-to-image generation tasks. 2. Some generated images, particularly those highlighted in Figure 10, exhibit unnatural aspect ratios and stretched elements, suggesting issues with the model’s handling of object proportions and spatial embeddings. 3. It would be great to discuss the scalability aspects, particularly how the proposed model handles graph sizes that exceed typical training configurations. 4. how the model performs with imperfect or noisy scene graphs, which are common in automatically extracted data.

Questions

1. The paper presents results on standard benchmarks. However, can you provide insights or preliminary results on how the model performs across datasets with higher variability in object complexity and scene density? 2. Why were certain closely related baselines omitted from quantitative comparisons? Could inclusion of these baselines provide a more comprehensive evaluation?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the challenge of attribute leakage in overlapping objects, which affects image fidelity in scenes with dense object interactions. While mitigation strategies are discussed via the CMA mechanism, further refinement is required to eliminate this issue completely. Additional exploration into the computational efficiency and scalability of the proposed methods would also benefit the paper, providing a more comprehensive view of their practical applications and limitations.

Summary

This paper proposes a method that uses a scene graph and integrates variational autoencoders (VAEs) and diffusion models to address complex scene generation. Specifically, a Semantics-Layout Variational AutoEncoder (SL-VAE) is used to derive diverse layouts and semantics from the scene graph, while a Compositional Masked Attention (CMA) combined with a diffusion model incorporates these elements as guidance. Additionally, a Multi-Layered Sampler (MLS) is introduced for isolated image editing. Experiments show that this approach outperforms recent methods in generation rationality and controllability.

Strengths

1. This paper considers an important issue in text-to-image generation realm. 2. The structure design in Section 3 makes sense. 3. The experimental results shown in Table 1 and 2 show the effectiveness of this method.

Weaknesses

1. My main concern is the practical application of this method. As we all known, scene graph building is not a trivial task, but you don't explain detail in the paper how to construct an exact scene graph. In addition, during inference, the prompt proposed by uses may be non-standard so that building a scene graph may be more difficult. 2. Besides, recent SOTA models, e.g. DALLE3, stable diffusion 3 try to solve the complex generation task by large-scale fine-grained dataset construction. How do you compare your methods with these data-centric methods. The authors should spend more space discussing the issues.

Questions

Please see the weakness

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Please see the weakness

Official Comment from Authors

Thanks for the authors' rebuttal. They have addressed my concerns. Thus, I would like to raise my score to 6.