BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/.

Paper

Similar papers

Peer review

Reviewer wCQb6/10 · confidence 4/52023-07-03

Summary

This paper proposes BLIP-Diffusion, a new subject-driven image generation model with multimodal encoder that supports multimodal control which consumes inputs of subject images and text prompts.

Strengths

The proposed method is novel and enables subject-driven generation under efficient fine-tuning and zero-shot setups. The proposed model can serve as a foundation model and combined with previous methods such as prompt-to-prompt and controlnet. Solid experimental results and ablations.

Weaknesses

The contribution is a bit lacking. It seems to me that the work is just a combination and the multimodal encoder from CLIP and the t2i model. The improvement under efficient fine-tuning setup seems to be incremental.

Questions

I am curious about the contribution of the proposed pretraining approach. How's the performance for the naive pretraining strategy (keep the same image as both input to the multimodal encoder and output to the diffusion mode.)? How do you get the random background image for generating synthetic image? For finetuning compared with Dreambooth, does it take the same time for each timestep? What's the inference time cost of the proposed method compared with standard stable diffusion?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Yes.

Reviewer tkfj8/10 · confidence 5/52023-07-03

Summary

This paper aims to solve the subject-driven text-to-image generation with a pre-trained subject representation that is derived from a vision-language encoder, the BLIP2 model. The obtained subject representation captures rich information of the visual input while being aligned with the textual space. The text-conditioned diffusion model is pre-trained to produce images based on this subject representation and the text input. The derived diffusion model can produce novel impressive renditions of a given subject with respect to different text prompts, in both zero-shot and few-step finetuned scenarios. Beyond that, the paper presents fancy applications like controlled generations and subject-driven stylization. Compared to the state-of-the-art approaches, the proposed BLIP-Diffusion demonstrates preferable subject-driven generation results both qualitatively and quantitatively while offering significant speed-up.

Strengths

The paper explicitly leverages the text-aligned image feature and the diffusion model trained on this subject representation is able to generate subject renditions with compelling quality. The novel combination with the BLIP2 model forms an effective solution to subject-driven text-to-image generation. The method presented is neat and elegant. Moreover, I like many technical details authors propose to achieve the final quality. For example, authors carefully design a synthetic procedure to form the synthetic training pairs to tackle the tendency to produce trivial solutions. Also, it is found that randomly dropping the subject prompt with some probability is beneficial to text-to-image generation. The paper demonstrates fancy applications in various applications and presents impressive generation results. the significant speed-up for the subject-driven generation makes it promising in practical usage. The quantitative study further proves the superiority of the method over prior leading approaches.

Weaknesses

- One possible way to explain the method is that this paper uses a detailed text prompt instead of a rough description like "an image of [V]" as used in Dreambooth. That is, the features captured from the BLIP2 may not be the key. One simpler baseline is "Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models". The paper should mention such baseline and add more discussions regarding this. - To avoid the distraction of the background, the authors propose to replace the background of the target image with random background. Then why not just remove the background during training? - One primary usage of subject-driven text-to-image models is to generate images related to humans. However, the paper avoids this scenario by explicitly removing human-related images during training, why is that? It is suggested to have a discussion regarding this, otherwise, it is suspicious that the method does not perform well on portraits. - While the paper qualitatively measures the fidelity to the input image and the text prompt respectively, there is no measure for image quality. While in the chosen samples of Figure 6 the BLIP-Diffusion shows apparent quality advantage over prior arts, it still needs a quantitative measure to reflect the image quality on a large amount of image results.T

Questions

It is suggested to provide more technical details for the style transfer part. Does it need some special prompt design when specifying styles according to the input image?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The limitation is sufficiently discussed in the supplementary material.

Reviewer ZD8x5/10 · confidence 4/52023-07-07

Summary

The paper introduces "BLIP-Diffusion", a new subject-driven image generation model that supports multimodal control using subject images and text prompts. The model introduces a pre-trained multimodal encoder to provide subject representation and enables zero-shot subject-driven generation and efficient fine-tuning for customized subjects. This model can be combined with existing techniques to enable novel subject-driven generation and editing applications.

Strengths

1. Originality: The paper introduces a novel subject-driven image generation model, BLIP-Diffusion, which supports multimodal control using subject images and text prompts. This model is original in its approach as it combines a pre-trained multimodal encoder for subject representation, enabling zero-shot subject-driven generation and efficient fine-tuning for customized subjects. 2. Quality: The quality of the paper is evident in the detailed explanation of the model and the comprehensive experiments conducted to validate its performance. The paper includes qualitative results that demonstrate the model's capabilities, such as zero-shot subject-driven generation and high-fidelity fine-tuning. The model also shows high subject fidelity and prompt relevance, requiring significantly fewer fine-tuning steps compared to other methods. 3. Clarity: The paper is well-structured and clear in its presentation. The authors provide a thorough explanation of the model, its implementation, and the experiments conducted. The use of figures and tables further enhances the clarity of the paper, providing visual representations of the model's performance and capabilities. 4. Significance: The significance of the paper lies in its contribution to the field of image generation. The BLIP-Diffusion model presents a new approach to subject-driven image generation, offering potential for novel subject-driven generation and editing applications. The model's ability to perform zero-shot subject-driven generation and efficient fine-tuning for customized subjects is a significant advancement in this field.

Weaknesses

Limited Zero-Shot Performance and Dependence on Fine-Tuning: The paper claims that the proposed BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Both qualitatively and quantitatively, the zero-shot results are not as impressive as one might expect. Furthermore, the model's performance seems to heavily rely on fine-tuning. While fine-tuning is a common practice in machine learning, the extent to which the model depends on it raises questions about its practicality and efficiency. The necessity of fine-tuning to achieve good results could be seen as a limitation, especially in scenarios where rapid or on-the-fly generation is required. This dependence on fine-tuning could limit the model's applicability and ease of use in certain contexts.

Questions

1. Clarification on Zero-Shot Performance: The paper claims that the BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Could the authors provide more evidence or examples to support this claim? 2. Editability Issue: The paper suggests that using trained background replaced subject images can address the editability issue. However, it seems that this approach might only deal with recontextualization, not subject area editing. Could the authors justify this approach and explain how it addresses the editability issue?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The paper's main claim is that the proposed BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Both qualitatively and quantitatively, the zero-shot results are not as impressive as one might expect. Furthermore, the model's performance seems to heavily rely on fine-tuning. While fine-tuning is a common practice in machine learning, the extent to which the model depends on it raises questions about its practicality and efficiency. The necessity of fine-tuning to achieve good results could be seen as a limitation, especially in scenarios where rapid or on-the-fly generation is required. This dependence on fine-tuning could limit the model's applicability and ease of use in certain contexts.

Reviewer dpqw6/10 · confidence 5/52023-07-07

Summary

The paper addresses the issues of lengthy fine-tuning and preserving the subject fidelity in subject-driven text-to-image generation models. Different from existing models such as Textual Inversion and Dreambooth that invert subject visuals into text embedding space, the paper introduces a new multimodal encoder which is pre-trained to produce visual representation aligned with the text. New subject renditions are then generated using such visual representation. The paper highlights the utility of the proposed approach in zero-shot subject-driven generation and editing applications, and demonstrates 20x speedup in fine-tuning for customized subjects.

Strengths

The paper is well-motivated and well-written. The key idea of the work to deeply align subject embedding and the text embedding is interesting. The proposed model is compatible with ControlNet and Prompt-to-Prompt, which has potential to unleash several important editing capabilities. The experiment results are solid, with sufficient evaluations.

Weaknesses

I see the key novelty of the proposed approach in section 3.2 where the paper introduces the subject representation learning. The paper mentions that output of the BLIP-2 multimodal encoder is passed to CLIP Text Encoder by combining the text and subject embeddings. I suggest authors elaborate this to provide more concrete details. We get embeddings as output from CLIP Text Encoder right? How is it possible to combine them before passing as input to CLIP Text Encoder. I might be missing something here, would be great if the paper clarifies this in rebuttal. Also, it is not clear if the paper additionally finetunes the CLIP Text encoder. In lines 164-164, it is mentioned that the Text encoder is also fine-tuned. But wouldn’t that bring language drift issues? Are there any specific strategies used by the paper in fine-tuning text encoder that aid in avoiding the language drift problem? Please clarify. Also, in section 3.2, how many synthetic pairs are used? I am happy to revise my final rating based on the clarifications in the rebuttal.

Questions

Please see weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors adequately addressed the limitations.

Reviewer dpqw2023-08-15

Thanks for the detailed responses to my questions on combining text and subject embeddings, language drift issues, and clarifying the number of synthetic pairs used. I updated my reviews.

Authorsrebuttal2023-08-14

Welcome further response and discussion

We appreciate the valuable reviews we receive, which help significantly to improve the manuscript. We have provided response to reviewers' questions and are committed to address remaining concerns. Please kindly let us know if there is additional feedback and suggestions.

Authorsrebuttal2023-08-15

Hi reviewer, we appreciate your time and effort in providing reviews and we have provided rebuttal accordingly. Does the rebuttal address your concerns? Please kindly let us know for remaining feedback.

Authorsrebuttal2023-08-15

Hi reviewer, we appreciate your time and effort in providing reviews and we have provided rebuttal accordingly. Does the rebuttal address your concerns? Please kindly let us know for remaining feedback.

Reviewer wCQb2023-08-17

Thanks a lot for authors' detailed responses. My concerns are mostly addressed. But I am still not quit sure about the efficiency improvement. As said, the "Subject embeddings are pre-computed" and the model takes the same time for each iteration. It is unclear to me why the model takes much less steps to fine-tune. Is the comparison fair? Are you using the same backbone as dreambooth? Now I keep my original score.

Authorsrebuttal2023-08-17

Thanks for your question and please kindly find our response below. As the title highlights, the improved fine-tuning efficiency comes from the “pre-trained subject representation”. This was motivated by the inefficiency of DreamBooth as described in LN 27, which we reiterate here: “*We attribute such inefficiency to the fact that most pre-trained text-to-image models do not natively support multimodal control - using both images and texts as control input. As a result, it becomes challenging to learn subject representation that aligns with the text space while capturing the subject visuals with high fidelity*”. Technically, DreamBooth always initializes fine-tuning **from a randomly initialized embedding**. In contrast, our model intializes fine-tuning **from subject embeddings produced by multimodal encoder**, which already capture largely the subject visuals even before the fine-tuning (evidenced by the zero-shot results). This results in a much smaller visual gap to learn during fine-tuning, which translates to less fine-tuning steps required. And yes, we use Stable Diffusion as described in LN 217. The DreamBooth model we compared to are with the same backbone. Definitely please let us know if more clarification is required and we are more than happy to further address remaining concerns. Thanks.

Authorsrebuttal2023-08-18

Dear reviewer wCQb, please kindly let us know whether the response addresses your concerns. Your feedback is greatly valued. Thanks.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC