Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models

Summary

The paper addresses the challenges of using large language models (LLMs) as prompt encoders in text-to-image diffusion models. It identifies two primary issues: the misalignment between LLM training objectives and the requirements of discriminative prompt features in diffusion models, and the positional bias introduced by the decoder-only architecture of LLMs. To tackle these issues, the authors propose a novel framework called LLM-infused Diffuser, which leverages human instructions and linguistic token refiners to enhance text representation capabilities. They design an LLM-Infused Diffusion Transformer (LI-DiT) and demonstrate its superior performance over state-of-the-art models in both open-source and commercial systems.

Strengths

* Introduction of LLM-infused Diffuser to facilitate the integration of LLMs into diffusion models to boost generation performance. * Experiments validate the effectiveness of this proposed framework when compared with both SOTA open- and closed-source baselines.

Weaknesses

* Scalability: While the paper demonstrates the superior performance of LI-DiT, the scalability of the approach to other diffusion models is not fully discussed and validated. * Lack of training and inference costs: Despite the superior generation quality, the paper does not provide detailed information on the training and inference costs associated with the proposed model.

Questions

* Could the authors provide information on both the training and inference costs associated with the proposed method, such as GPU memory consumption, training time, and inference time? * The paper mentions that the LLM-infused Diffuser can be easily and flexibly integrated into diffusion models. Does this imply that once trained, the LLM-infused Diffuser can seamlessly integrate into existing diffusion models without further fine-tuning? Besides, do diffusion models require additional fine-tuning to effectively cooperate with the LLM-infused Diffuser? It would be better to see results or insights into integrating the LLM-infused Diffuser into other diffusion models.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper presents an investigation into the integration of Large Language Models (LLMs) into text-to-image diffusion models. It identifies issues with using LLMs as prompt encoders, namely misalignment between next-token prediction training in LLMs and the need for discriminative prompt features in diffusion models, as well as positional bias introduced by the decoder-only architecture. The authors propose a new framework to overcome these challenges and introduce an LLM-Infused Diffusion Transformer (LI-DiT) to leverage LLMs effectively in image generation tasks. The paper also discusses broader societal impacts and adheres to ethical guidelines.

Strengths

- The proposed method addresses a significant gap in utilizing LLMs for prompt encoding in diffusion models, offering a new solution to enhance text-to-image generation. - The paper includes a discussion on potential societal impacts, considering both positive and negative outcomes,

Weaknesses

- The visual results of I2T may exhibit cherry-picking, including Figure 1 and Figure 7. The authors need to provide further explanations. Although they claim there are more prompt-generated results in the appendix, this clearly does not mitigate the risk of cherry-picking. - Integrating LLMs into diffusion models is not uncommon, and the authors lack discussion and analysis of such works, e.g., [1][2][3]. More in-depth exploration by the authors is needed. - Honestly, in the visual results of Figure 7, I did not notice any particular advantages of the proposed method. - Besides the visual effects, the proposed method does not seem to offer significant improvements over existing methods and lacks theoretical support. There is a need for further improvement in novelty. [1] LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation [2] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion [3] SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models

Questions

see weakness

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

In the context of text-to-image (T2I) generation, this work addresses the problem of leveraging representations from state-of-the-art decoder-based LLMs for conditioning image generation. The authors highlight challenges of leveraging existing LLMs, namely - misalignment in representations due to differing training objectives (autoregressive v/s non-autoregressive) and the effects of positional bias. The authors then propose ways to mitigate this misalignment via prompt changes (leveraging instruction tuning of LLMs), embedding ensembling techniques (for positional bias), and a new DiT based on this ensembled conditioning mechanism. The proposed observations and method show consistent qualitative and quantitative improvements on benchmarks across the board.

Strengths

This paper addresses an important and relevant problem of aligning a) very capable LLM models with b) capable image generation models and understanding why and if there could be a mis-alignment between these two objectives. I like the motivation of leveraging instruction tuned language models and it is understandable why one might want to switch to, or at least explore these models for conditioning text to image generation. Further, the results are impressive and I appreciate the ablation studies conducted for different components proposed in the work, and the custom benchmark to probe the alignment. The paper is presented in a clear and concise manner with appropriate visuals where necessary.

Weaknesses

While the paper shows significant empirical improvements, I have some concerns and questions where it would be great to have a discussion with the authors: 1. Image generation with diffusion models is non autoregressive, while decoder-LLM based representations are trained with auto-regressive models. There is much needed discussion surrounding the choice of models (Encoder decoder, encoder only, decoder only) which does not seem to be addressed in this work. For example, it seems that encoder-only representations are a natural choice for this task (such as InstructOR[1] embeddings, which are tuned for multi task learning in an encoder-only setup, or others from the MTEB [2] benchmark). How do they, or rather would they perform? What is the intuition? 2. The motivation behind adding ensembling, connections with positional bias, and the linguistic token refiner from different LLMs seems lacking. What is intuition here? Why does one need “..The image prompt with instructions to be encoded by multiple frozen LLMs separately”? 3. Re. the positional bias: Given that the LLMs tested in the paper are still small (~2-7B), it may be possible that their “long context” understanding may be limited as an artifact of their instruction tuning. Thus, is the positional bias inherent to these decoder based LLMs or a capability drawback of the smaller LLM on long context itself? That is, can this gap be closed if one employs an LLM with impressive and uniform long context capabilities? [1] One Embedder, Any Task: Instruction-Finetuned Text Embeddings Su, Hongjin, et al. [2] Muennighoff, Niklas, et al. "MTEB: Massive text embedding benchmark." Suggestions: 1. I would encourage the authors to further expand on related work, especially focussing on the alignment between language model types and the task of (non-autoregressive) image generation. 2. (Presentation) The coherence between claims can be improved (it seems that many observations -> design changes) are grouped together.

Questions

(Besides the ones in the above section) In the experiments, what is the average prompt context length (min, max, avg)?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Discussed.

Summary

This work identifies two main reasons for degraded prompt-following ability in image generation with decoder-only transformers: the misalignment between pretraining objective and diffusion's need of discriminative prompt feature, as well as the intrinsic positional bias for decoder-only transformer. The solution is to enhance text representation capability and remove positional bias. This work also proposes a diffusion architecture conditioned on multiple LLMs. The proposed LLM-Infused Diffusion Transformer (LI-DiT) surpasses previous state-of-the-art open-source models in prompt understanding.

Strengths

* Analysis of the reasons that LLMs not working out-of-the-box is interesting. * The qualitative and quantitative results demonstrate advantages over prior baselines. * The ablation studies are solid. The contribution of each contribution is demonstrated. * The paper is easy to read and understand.

Weaknesses

The main weakness of this work is its lacked novelty. * The Input Prompt part (Sec 3.1) is prompt engineering. * The Linguistic Token Refiner (Sec 3.1) can be considered as adding more transformer blocks that are trainable and have full-attention, with the LLM weights frozen. * The cross-attention blocks in Collaborative Refiner (Sec 3.1) can be considered as self-attention of concatenated features without attending to each other within the same LLM tokens. * The technical contributions in Sec 3.1 do not exhibit significant novelty. Furthermore, the comparisons with other works are not clear. * What the sizes of the circles indicates in Fig. 2 is not clear. * The model sizes of the baseline models under comparison are not listed in Tab. 1. Finally, this work ignores an existing line of work that combines LLMs with diffusion models [1,2,3]. These works use LLM to generate intermediate representation generation and then generate images conditioned on the intermediate representation. [1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Feng, et al. NeurIPS 2023. [2] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. Lian, et al. TMLR 2024. [3] Grounded Text-to-Image Synthesis with Attention Refocusing. Phung, et al. CVPR 2024.

Questions

The reviewer's questions are primarily from the weaknesses section, specifically: 1. What are the sizes of the circles indicate in Fig. 2? 2. What are the model sizes and the training dataset size of the models in Tab. 1? 3. What is the performance of a model if the collaborative refiner is replaced by the simple linguistic token refiner, with input concatenated in the sequence length dimension?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The reviewer did not find unaddressed potential negative societal impact. The unaddressed limitations are described in the weakness section.

What's the connection between the cases shown in Figure 4 and the textual hidden state we actually need for DiT?

Hi, thanks for the great work. I notice the cases shown in Figure 4, where the causal LLM generates the ongoing text while T5 repeats the text. Denote the text as $X$, when we use LLM as a textual information extractor, we usually directly treat the hidden state of $X$ from the last layer of LLM (denote as $f_\theta (X)$) as the representation, and feed $f_\theta (X)$ to DiT model for image generation. I have a question, the case in Figure 4 shows the continual generation process of LLM (given $X$ -> predict $Y$) rather than the feature of the representation $f_\theta (X)$. We just need to use $f_\theta (X)$ for DiT rather than $f_\theta (Y)$. What's the connection between the cases shown in Figure 4 and the hidden state we actually need? I hope the authors can answer this question, many thanks!