RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models

Summary

This paper proposes to balance the realism and compositionality of the generated images by means of different diffusion models, such as pre-trained T2I models and models based on spatial perceptual control. The authors develop a balancer that optimizes the two coefficients through mask-based attention manipulation to enable dynamic combination of the different models. They also demonstrate the effectiveness of the method through extensive experiments.

Strengths

Pros: 1. The proposed design could be used to bridge different models and support better image generation. 2. SOTA quantitative and qualitative results obtained by the proposed method. 3. The method can be extended to stylized image generation in a training-free manner.

Weaknesses

Cons: 1. Unclear Task definition. First, even different pre-trained T2I models can result in different so-called REALISM due to different training strategies and data. Secondly, Control-based models, e.g., Controlnet and GLIGEN, all add controllability while inheriting the realism generated by the original T2I models. So I'm not sure why the authors need to propose another method to do the so-called balancing of realism and composability.In addition, how can you guarantee that the combination of T2I and L2I can generate real diagrams when the images generated by T2I are all more fake? 2. Unconvincing observation. In Figure 1 and paragraphs 2 to 4, the authors present their motivation only through an existing method, GLIGEN. I argue that this is inappropriate. After all, there are many methods based on layout or control, no matter what conditions these methods use, such as layout, segmentation map, and pose. Motivation is the key to a paper and cannot be deduced from just one method. So I argue that this observation is not very convincing to me. 3. Insufficient contribution. It is common to use optimization to produce better images, but the difference lies in the design of the optimization. The use of mask to combine with attention to do optimization is more common way, such as in [1,2,3,4,5]. The only difference is that the existing methods may optimize zt, this paper optimizes two coefficients. So I argue that the contribution of this paper is not very sufficient.

Questions

1. Fig-1 lacks a detailed explanation. 2. Sub-sec 3.3 contains not much valuable information, as the only difference between Eq.9 and Eq.6 is how to obtain a mask/spatial condition. I will revise my rating according to the author's feedback and the reviewer's discussion.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

1

Limitations

N/A

Summary

This paper presents a method, named RealCompo, for combining multiple diffusion models, such as a text-to-image model and a spatial-aware one, to achieve the best of both worlds: superior image realism and compositionality. It merges predicted noise from both models during each denoising step, and balances them based on two sets of coefficients that are dynamically updated through a loss that forces cross attention maps to follow provided spatial constraints. Experiments show that RealCompo improves both realism and compositionality compared to using either model alone, and it outperforms SOTA spatial-aware models in compositional generation.

Strengths

- The paper is well written and easy to follow. - The proposed method is straightforward and compatible across various models. - The idea of fusing multiple diffusion models during denoising is interesting. This may have applications beyond compositional text-to-image generation. - Extensive experiments are conducted on various T2I models, spatial-aware models, stylized models.

Weaknesses

- Since the proposed method runs multiple models concurrently and requires gradient-based updates, a discussion on computational efficiency is crucial for its practical application. - Figure 1 does not illustrate the trade-off between realism and compositionality well. I don't see an apparent drop of realism/aesthetics as $\beta$ (control of layout) increases.

Questions

- Why can RealCompo improve image realism (Table 2, Figure 6 left) and style preservation (Figure 6 right) beyond the capabilities of the T2I model alone? This is difficult to understand because the loss function (Eq. 6) employed for balancing the two models is designed solely to encourage better adherence to the provided spatial constraints. I will reconsider my rating if this aspect can be properly addressed. - What is the purpose of initializing the coefficients as random values (Eq. 1)? This seems unnecessary and introduces random bias towards pixels.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors touched on the computational cost in Appendix B.5. However, a more detailed comparison with T2I and L2I models concerning computational overhead and memory overhead is crucial to determine the feasibility of the proposed method.

Summary

The paper introduces a training-free and flexible text-to-image generation framework called RealCompo, which enhances compositional text-to-image generation by balancing the realism and compositionality of generated images. It features a novel balancer that dynamically combines the predicted noise from T2I models and spatial-aware image diffusion models (such as layout, keypoint, segmentation map). This framework provides a fresh perspective for compositional image generation.

Strengths

1. The proposed method is training-free but achieves notable results. 2. The proposed method mainly applies a balance between two pretrained diffusion models, which is easy to implement and quite flexible. 3. The paper’s explanation of realism and control sounds reasonable.

Weaknesses

1. The biggest problem is novelty/contribution. The paper uses a balancer to balance two diffusion models, which is very similar to the concept of "checkpoint merge" / "model ensemble". It is necessary to clarify the differences between these two approaches.

Questions

I will consider increasing the score or maintaining it if the authors can address the following issues. 1. The paper uses a balancer to balance two diffusion models, which is very similar to the concept of "checkpoint merge" or "model ensemble". It is necessary to clarify the differences between these two approaches. 2. Please explain the necessity of the proposed method. If condition-guided models like ControlNet use a stronger backbone, higher quality, larger quantity, and more diverse data and annotations during training, is it still necessary to balance with non-condition-guided text-to-image models?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

A minor clarify

Dear Esteemed Reviewer XVVX, We would like to clarify that the supplement PDF refers to **the pdf in global response**. We apologize for any confusion this may have caused. Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly. Thank you once again for your invaluable contribution to our research. Warm Regards, The Authors

A minor clarify

Dear Esteemed Reviewer ZHfz, We would like to clarify that the supplement PDF refers to **the pdf in global response**. We apologize for any confusion this may have caused. Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly. Thank you once again for your invaluable contribution to our research. Warm Regards, The Authors