Novel Object Synthesis via Adaptive Text-Image Harmony

Summary

The paper works on novel object synthesis, namely, given an image and text prompt, the proposed method can generate a new image that contains the visual features from the given image and the textual information from the given prompt. The paper proposed a method names Adaptive Text-Image Harmony to tackle the task. The authors provide both qualitative and quantitate comparisons to show the effectiveness of the method.

Strengths

The paper is easy to follow. The author proposed an effective method that solves the challenging problem of novel object synthesis. The paper provides extensive experiments, along with human evaluation, to justify the effectiveness of the approach. Authors also provide detailed ablation for the design of the approach.

Weaknesses

For the constructed dataset with 1,800 text-image pairs, will the authors release it? Could this dataset be used to fine-tune other baseline approaches to improve performance? If so, has this experiments been done? For the baseline methods, e.g., InstructPix2Pix, did the authors directly use the released model or train a new one using the same text-to-image model as this work, which is SDXL-Turbo?

Questions

Please refer to the weakness section.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper aims to generate novel objects based on a reference image and a conditional text prompt (e.g., a bottle (image) with a penguin (text) outlook). To this end, a method called Adaptive Text-Image Harmony (ATIH) is proposed to better align the conditional image and text. Experiments show that ATIH yields better results compared to existing state-of-the-art methods.

Strengths

- Overall, the paper is easy to follow. - Experiments in the main paper and supplementary materials try to be as thorough as possible to cover the most promising frameworks (e.g., ConceptLab) and ablation studies. - On a high-level idea of fine-grained controllable image generation, I think this topic will attract interest in the image editing/image manipulation community. - While the idea of manipulating attention layers of diffusion models is not new, I think this paper finds a way to (1) preserve the structure of the reference image as much as possible while (2) editing the image toward the given text -- which I consider a main contribution of this paper.

Weaknesses

- As far as I understand, the image for text information in Figure 2 (e.g., "iron", "white shark") is for visualization purposes only. Actually, I think this might mislead readers (e.g., models can take two reference images as input). - I think one thing the authors must consider is adjusting the strengths of existing baselines (e.g., InstructPix2Pix). If baselines allow for controlling the strength of the edit, it would be fairer if we had comparisons to that (e.g., similar to Figure 16, but for baselines). - While in the main paper, the Golden Section Search algorithm seems to play an important role, it's unclear how important this algorithm is (e.g., can we simply replace this with simple average values). In other words, I think the current paper lacks an ablation study for this module.

Questions

It'd be interesting if the authors can discuss how to make a better evaluation for this task. Overall, I think this task is ambiguous and difficult to evaluate (e.g., if you want to turn a cat into a peacock, is it better to (1) completely replace the cat with the peacock, or (2) yield a combination of the cat and the peacock?) How can we define which is a 'better alignment' with the user's intention? (This is an open-ended question and will not affect my recommendation for this paper).

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors did briefly discuss the limitations in the Supplementary. At this moment, I'd recommend this paper for acceptance, as I believe the task introduced by this paper outweighs the limitations in experiments and ablation studies.

Summary

The paper proposes a new method for harmonized image generation conditioned on both text and image conditions, leading to better performance than baselines mentioned in the paper.

Strengths

The idea of using scale factor to combine different conditions is straightforward and reasonable. It is also The qualitative results shows better image fidelity compared to some previous image editing methods.

Weaknesses

The optimal combination between image and text condition is subjective. In the paper, all the generate examples have similar depth/layout/structure with input image condition, while with modified texture/appearance guided by the text. Assume the user want to generate "an image of a shark" with color/pattern similar to a given dog image, i.e. which is different from the example in Figure 2. It is not clear whether the proposed method can achieve that. Furthermore, some important baselines are missing: 1. The depth/structure information from the input image can be extracted and injected into the generation process with a ControlNet-like model [1]. This baseline is important because there is little structural change shown in the results of proposed methods, which is the scenario that the well-known ControlNet is good at; 2. A naive way to generate with two conditions is using revised classifier-free guidance as [2]. The guidance strength from image and text can be controlled by hyper-parameters. 3. Another naive baseline is to only use image guidance before timestep t , and use text guidance after t. Results with different t will be interesting. Subject-driven image generation and editing methods [3,4,5,6,7,8, 9] need to be discussed or at least mentioned, as these methods also aim to generate image guided by the reference images and text prompts. [1]. Adding Conditional Control to Text-to-Image Diffusion Models. Lvmin Zhang, Anyi Rao, Maneesh Agrawala. [2]. InstructPix2Pix: Learning to Follow Image Editing Instructions. Tim Brooks, Aleksander Holynski, Alexei A. Efros. [3]. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. [4]. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. Nataniel Ruiz Yuanzhen Li Varun Jampani Yael Pritch Michael Rubinstein Kfir Aberman. [5]. Subject-driven Text-to-Image Generation via Apprenticeship Learning. Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen. [6]. Customization Assistant for Text-to-image Generation. Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun. [7]. Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning. Jian Ma, Junhao Liang, Chen Chen, Haonan Lu. [8]. Multi-Concept Customization of Text-to-Image Diffusion. Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu. [9]. Kosmos-G: Generating Images in Context with Multimodal Large Language Models. Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei.

Questions

Can the proposed method generate object whose shape is guided by the text prompt, while appearance/texture is guided by a reference image?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper introduces an innovative method to generate new object images by combining textual descriptions with corresponding images. Addressing the common imbalance in diffusion models between text and image inputs, the authors propose the Adaptive Text-Image Harmony (ATIH) method. This method balances text and image features during cross-attention, ensuring a harmonious integration. Key to this method are the scale factor (α) and injection step (i), which adjust the influence of text features and preserve image information. A novel similarity score function maximizes and balances the similarities between the generated image and the input text/image, while a balanced loss function with a noise parameter optimizes the trade-off between editability and fidelity of the synthesized image. Validated through extensive experiments on datasets like PIE-bench and ImageNet, ATIH outperforms state-of-the-art techniques in creative object synthesis. The method showcases remarkable results, such as creating a dog-lobster or a rooster with an iron texture, demonstrating its innovative and high-quality image generation capabilities. The framework includes a Text-Image Diffusion Model (TIDM) using a pre-trained SDXL Turbo model with dual denoising branches. By treating sampling noise as a learnable parameter and designing a balance loss function, the method enhances image fidelity and editability. The ATIH method adaptively adjusts the scale factor and injection step to balance text and image similarities, using a similarity score function and the Golden Section Search algorithm to find optimal parameters.

Strengths

1. Originality: The paper introduces a novel method for combining textual descriptions with corresponding images to generate new object images. The originality lies in the Adaptive Text-Image Harmony (ATIH) method, which effectively balances text and image features during the cross-attention mechanism in diffusion models. This innovative approach addresses a significant challenge in existing methods— the imbalance between text and image inputs— and offers a robust solution to achieve harmonious integration. The introduction of a scale factor (α) and an injection step (i) to fine-tune this balance further highlights the creativity and novelty of the approach. 2. Quality: The paper demonstrates high quality through comprehensive experiments and rigorous validation. The authors provide detailed methodological explanations and present clear experimental results on datasets such as PIE-bench and ImageNet. The inclusion of a balanced loss function with a noise parameter to optimize the trade-off between editability and fidelity showcases the thoroughness of the approach. The comparison with state-of-the-art techniques further establishes the effectiveness and superiority of the ATIH method. The experimental results, including examples like a dog-lobster and a rooster with iron texture, illustrate the method's capability to produce high-quality, innovative, and coherent images. 3. Clarity: The paper is well-written and clearly structured. The authors provide a concise and comprehensive introduction to the problem, followed by a detailed explanation of the proposed ATIH method. The use of figures and diagrams, such as the framework of the Text-Image Diffusion Model (TIDM) and the results of the experiments, aids in understanding the methodology and its impact. The step-by-step breakdown of the methodology, along with the clear presentation of experimental results, ensures that readers can easily follow the proposed approach and its benefits. The balanced loss function and similarity score function are well-explained, contributing to the overall clarity of the paper. 4. Significance: The proposed method's capability to outperform existing techniques and produce high-quality, innovative results underscores its importance and impact on the broader field of text-to-image synthesis.

Weaknesses

1. While the paper discusses a novel algorithm for text-image harmony, the experimental validation focuses heavily on visual outcomes without substantial quantitative backing or comparisons against baselines using metrics relevant to the text-image synthesis field. Including more diverse and robust statistical evaluations could strengthen the claims about the algorithm's effectiveness. 2. The experiments primarily utilize a specific set of images and texts, which might not adequately represent the variety of real-world scenarios where such an algorithm could be applied. Expanding the dataset to include a wider range of text and image pairs, especially challenging ones, would help in understanding the algorithm's limitations and strengths better. 3. The paper could benefit from a deeper exploration of the robustness of the proposed method, particularly in how it handles edge cases or unusual text-image pairs. This includes testing the method's performance on non-ideal or adversarial inputs to gauge its resilience and adaptability. 4. More detailed comparative analysis with existing methods, especially recent advancements in text-to-image synthesis and image editing technologies, would provide a clearer picture of where the proposed method stands in terms of innovation and improvement. Specific examples of where it outperforms and underperforms can guide future development. 5. The methodology section could be expanded to include more technical details about the implementation, which would aid in reproducibility. For instance, details on parameter settings, algorithmic steps that were particularly effective, and potential pitfalls in the implementation could be highlighted. 6. The paper mentions user studies but does not delve deeply into how the feedback from these studies was integrated into the algorithm refinement. Elaborating on this process could provide insights into the user-centric development of the algorithm and its practical usability.

Questions

1. Please include a detailed description of the selection criteria for the text-image pairs used in your experiments. It is important to demonstrate the diversity and representativeness of these pairs to ensure the robustness of your Adaptive Text-Image Harmony (ATIH) method. 2. It would be beneficial to expand the comparative analysis section with more detailed results, including statistical significance tests. This would provide a clearer picture of how your method improves over existing methods, underlining its novelty and effectiveness. 3. Please discuss the performance of your method under non-ideal conditions, such as complex or abstract text descriptions. Identifying and describing limitations observed in such scenarios could guide future improvements and research. 4. Including comprehensive implementation details, particularly regarding parameter settings and platform-specific optimizations, would significantly aid in the reproducibility and independent verification of your results. 5. Identifying specific edge cases and failure modes encountered during testing and development would help in understanding the practical limitations of your model. This information is crucial for setting realistic expectations and guiding future research directions.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have not been addressed.

Response for authors

Thanks, I've reviewed the author's response. However, I still have several concerns regarding this paper. The primary contributions—a scale factor to balance text and image features in cross-attention and a novel similarity score function—while further clarified with specific parameter values, lack sufficient theoretical justification. Besides, the comparison of the proposed method with others, while claiming superiority in fusion quality, appears to rely on somewhat selective examples (e.g., African chameleon-bird). A broader range of comparative cases or more challenging scenarios would provide a more comprehensive understanding of the method's strengths and weaknesses.