Subject-driven Text-to-Image Generation via Apprenticeship Learning

Summary

This paper proposes a method for subject-driven text-to-image generation, where a model is tasked to generate novel renditions of a subject given a few images of that subject. Different from previous fine-tuning approaches, this paper trains a model that conditions its generation on the given subject images. To train such a model, this paper first trains a large amount of subject-specific fine-tuned models, and use these fine-tuned models to generate new training data for knowledge distillation. The distilled model achieves good qualitative performance.

Strengths

- It is nice that the apprenticeship model does not require finetuning on new subjects. It is an interesting idea to directly use subject images as the conditional input for generation. - The paper is mostly well-written. - The learned model achieves good qualitative results.

Weaknesses

- The training data covers a wide range of subjects. Therefore, it is hard to tell if the apprenticeship model learns to generalize to new subject or not. Have the authors performed de-duplication to ensure that the subjects used for evaluation do not appear in the training set of the expert models? - The proposed method requires training of 2M expert models, where each expert is a 2.1B Imagen model. This is extremely expensive both computational-wise and storage-wise. - The inference speed of the apprenticeship model seems to be slow, as each demonstration sample needs to pass through the Imagen model. Methods such as DreamBooth do not incur additional inference cost after the finetuning. - It would be nice to see some ablation on the importance of the expert model. For each subject cluster, how many synthetic images are used to train the apprenticeship model? What if only the real images are used to train the apprenticeship model? - It would be better if the paper could give a more detailed illustration of the apprenticeship model's architecture, instead of referring to the ReImagen paper.

Questions

- Is it possible to finetune the apprenticeship model on new subjects?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes the authors have addressed the limitations.

Summary

The paper presents a novel subject-driven text-to-image generator named SuTI. This model leverages in-context learning as opposed to subject-specific fine-tuning. SuTI is built upon the principles of apprenticeship learning and is capable of generating high-quality, customized, subject-specific images. Remarkably, it achieves this at a speed that is 20 times faster than optimization-based methods. SuTI has demonstrated superior performance over existing models on benchmark tests such as DreamBench and DreamBench-v2. The paper highlights the recent advancements in text-to-image generation models, which have shown significant progress in generating highly realistic, accurate, and diverse images from given text prompts.

Strengths

The paper exhibits several strengths across the dimensions of originality, quality, clarity, and significance: 1. Originality: The paper introduces SuTI, a novel subject-driven text-to-image generator that uses in-context learning instead of subject-specific fine-tuning. This approach is original and innovative, as it deviates from the conventional optimization-based methods, offering a faster and more efficient solution. 2. Quality: The quality of the paper is evident in the rigorous testing and validation of the SuTI model. The model has been benchmarked against existing models on DreamBench and DreamBench-v2, where it has shown superior performance. This demonstrates the robustness and reliability of the model. 3. Clarity: The paper is well-structured and clear in its presentation of the SuTI model. It provides a comprehensive explanation of the model's workings, its applications, and its performance in various tests. The use of visual aids and examples further enhances the clarity of the paper. 4. Significance: The significance of the paper lies in its contribution to the field of text-to-image generation. By introducing a faster and more efficient model, the paper pushes the boundaries of what is currently possible in this field. This could have far-reaching implications for a variety of applications, including content creation, design, and more.

Weaknesses

One weakness of the paper is the lack of discussion around the cost and complexity of constructing the training dataset. The process of creating a comprehensive and diverse dataset for training a model like SuTI can be a significant undertaking, both in terms of time and resources. The paper does not delve into the specifics of this process, leaving readers without a clear understanding of the potential challenges and costs associated with data collection and preparation. This lack of transparency may make it difficult for others to replicate the study or apply the model in different contexts.

Questions

1. Dataset Construction: Could you provide more details about the process of constructing the training dataset for SuTI? Specifically, how much time and resources were required to create the dataset? Were there any significant challenges encountered during this process? 2. Model Scalability: How scalable is the SuTI model with respect to the size and diversity of the training dataset? If the dataset were to be expanded or updated with new subjects or scenarios, would this significantly impact the model's performance or the resources required for training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper introduces SuTI for the subject-driven text-to-image (T2I) generation method. Numerous expert models are first trained on millions of image clusters collected from the internet, each focuses on a specific visual subject. A dataset is then created, consisting of concept images, target prompts, and corresponding target images. By training the apprentice model on this dataset, SuTI can generate high-quality and customized subject-specific images without the need for test time fine-tuning. Both the qualitative and quantitative experiments are performed with the baselines.

Strengths

- The presented method is well-motivated and easy to understand. - SuTI demonstrates fast generation and has a broad domain of applicability. - The paper includes extensive experimental evaluations with impressive results.

Weaknesses

Some details of the experiments require clarification: - The hyperparameters used for training the baselines, such as the number of training iterations and learning rates, should be provided. - More information is needed regarding the human evaluation, including the number of questions, the number of evaluated images, and the number of users involved.

Questions

- More details about the experiments should be clarified. - Can the proposed model perform multiple concept generation, which is also important in concept image generation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

The paper proposes an in-context learning method for model customization given personalized objects. The method first collects a large-scale dataset of custom concepts ensuring all images in each custom concept cluster are similar to each other and fine-tunes the model for each concept using Dreambooth. These expert models are then used to get the dataset to train the in-context learning method. It takes a few image text pair of the concept and a new text prompt to generate the image corresponding to the new prompt. The method is based on the Re-Imagen framework. To ensure high quality image text pair dataset from dreambooth models, CLIP based feature similarity threshold is applied.

Strengths

The method is one of the first works on in-context learning for model customization. This prevents the time-consuming step of fine-tuning models given any new subject images. Both qualitative and quantitative results show that the method performs on par or better than existing zero-shot or fine-tuning methods. The paper is well-written, easy to understand, and has extensive ablation experiments to validate the importance of different aspects of the method.

Weaknesses

Why is there a need to train the Dreambooth expert models for training the final SuTI model? How does the performance change if the collected dataset itself is used directly to train the final model with N-1 image text pairs for in-context input and 1 sample as the new inference? It would be great to have an analysis regarding that. Other objects included in the image often take the characteristics of the main subject, e.g., kite in Figure 10 2nd column last row or the british shorthair cat in Fig. 9 1st column last row in the appendix. Does adding a specific characterization in the text prompt for the other subject prevent that? Or is it overfitting in such scenarios? Does having more in-context demonstrations prevent that? Can it be combined with other editing-based methods like SD-Edit or similar approaches to edit a specific region of the image or combine multiple specific subjects in the same image? E.g., to generate the canine dog of Figure 5 eating the cherry bowl of Figure 4. The editing example shown in Figure 5 changes the whole image instead of just the birds in the TV show. Another point I would like to make is that it would have been great to also show the performance of SuTI with the SD backbone. This is not a weakness per se, and it doesn't affect the final rating. It's excellent work, but it would be helpful to assess the method's performance on such open-source models. Some of the baseline methods shown in the paper are already with the SD backbone.

Questions

Minor questions and comments: 1. Does having a variable number of demonstrations instead of 3 have any effect on the final model performance? 2. It would be nice to include some sample descriptive text predicted by the language model for various subjects. Is it usually a class word corresponding to the subject or includes more details regarding the subject? 3. Is there any reason for low photorealism score of SuTI compared to DreamBooth? 4. Line 94. "to to" typo 5. Line 116. "models" -> "model" 6. Line 209. Probably a missing word in the sentence. "with largest space consumption" 7. Line 280. "generated generated" repeated words. 8. Line 286. “GAN=based” -> “Gan-based” 9. Line 306. "subjects are contain" -> "subjects contain" 10. Appendix Line 429. "Subject-Drive" -> "Subject-Driven" 11. Appendix Line 430. "task of our paper aims to solve" -> "task our paper aims to solve"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

In this paper, the authors propose a new model that can perform subject-driven text-to-image generation. Instead of fine tuning a leerte pretrained model on each subject, the authors use apprentice learning to first construct a virtual dataset from a large number of teacher models, each specific to a kind of subjects, and then have the student model learn from the constructed dataset. The authors have also shown strong qualitative and quantitative results and ablation study.

Strengths

The paper is very well written and easy to follow. The results seem very promising. The paper tackles an interesting problem that is relevant to the applications.

Weaknesses

1. This paper could have been impactful if the authors plan to open source or provide a way to reproduce the results. However, from the checklist the authors indicate that they have no plan to open source the model. Given that this model is trained with hundreds of TPUs, unless the authors provide the pretrained model or explicit instructions on how to reproduce the results in a non-cost prohibitive way, I don’t really see a way for the peer researchers to verify the results, nor do I see any real benefits for the practitioners from this paper. 2. The authors performed human evaluations. However, details about the instructions given to human evaluators and the compensation are not provided.

Questions

Please address the weakness mentioned above. I am happy to change my score if problems related to my concerns are resolved. How long does it take to train the apprentice model?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

There is no discussion of the details about the human evaluation (I.e. the instructions given to human and the compensations).

About rebuttal

Thanks again for providing the detailed reviews! We are wondering whether you have read our rebuttal. If there is anything else we can discuss or clarify, please let us know. We are more than happy to discuss further during the author-reviewer discussion period.

Thank you for your response

Thank you for your response. Based on the prospective open source model, I will change my rating from rejection to acceptance. And I do want to comment that Comment #2 Since the model is extremely computationally expensive, even if the authors provide technical details in the paper, it is still very unlikely for general readers to reproduce the results without the open source model. Comment #3 I agree with the author rebuttal that the paper can provide some insight to the methodology. Thank you for your efforts to open source this model. Looking forward to its release.

Thanks for the response

The rebuttal addresses most of my concerns. Looking forward to the open-source model/API as well. Thanks. I will keep my current rating.