DomainGallery: Few-shot Domain-driven Image Generation by Attribute-centric Finetuning

Summary

This paper tackles the problem of few-shot image generation, in a scenario where it is domain-driven and defined by several attributes shared among different target data.

Strengths

1. The research topic is of general interest in the generative models, and this is a good bridge between subject-driven generation and the domain-driven generation, using few-shot training data. 2. The proposed method/framework, DomainGallery, is shown effective on different generation scenario. 3. The experiment results are somehow sufficient, good-looking, both quantitatively (FID,KID, i-LPIPS) and qualitatively (visualization in different setup).

Weaknesses

1. The base models covered in this work are limited. In fact, while Dreambooth is one of the exemplar baseline for few-shot text-to-image (T2I) models, there are more latest and advanced T2I models also suitable for this study. I would suggest the authors to cover 1-2 more advanced T2I models to demonstrate that the proposed DomainGallery framework is a generalized method and applicable to different model structure / design. 2. I did not see any discussion / visualization of the possible failure cases. It is better to include the failure examples to show the boundary of the proposed method.

Questions

1. I would like to ask if the authors consider the mixture of style/attributes, in contrast to the multi-subject customization work, using different unique identifiers? Indeed, I do see the mixture of [style] and [subject] results in Figure 6, but it is better if you can include the results of mixture-of-style as the supplement. 2. Are visualization results in the experiments obtained via fixed/deterministic sampling and fixed input/noise? In this way, we can compare different methods clearly and fairly.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discussed the limitations and broader impact properly in the appendix C and D. Meanwhile, I appreciate that the authors acknowledge the cherry-picking and issues of generated images on the Sunglasses dataset, as mentioned in the limitation section.

Summary

This paper introduces a few-shot domain-driven image generation method that finetunes pretrained Stable Diffusion in an attribute-centric manner. It features prior attribute erasure, attribute disentanglement, attribute regularization and attribute enhancement.

Strengths

1. The results seem good without overfitting. 2. Overall the paper is well-written and easy to follow.

Weaknesses

1. The similarity consistency loss has been utilized in GAN-based methods. It is similar to “Few-shot image generation via cross-domain correspondence”[1]. 2. The comparative experiments for personalization are missing, such as textual inversion. 3. The concept of multi-attribute transfer is quite similar to hybrid domain adaptation[2]. It needs to include a conceptual comparison in the related work section. > [1] Ojha U, Li Y, Lu J, et al. Few-shot image generation via cross-domain correspondence[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 10743-10752. > [2] Li H, Liu Y, Xia L, et al. Few-shot Hybrid Domain Adaptation of Image Generator[C]//The Twelfth International Conference on Learning Representations. 2023.

Questions

As seen in weakness.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The manuscript includes discussions on limitations and social impacts.

Summary

Text-to-image generation models pre-trained on large-scale datasets made progress but the models are still limited when we expect to generate images that fall into specific domain or style that are hard to describe or unseen to models. This paper proposed the DomainGallery, a few-shot domain-driven image generation method which aims at finetuning pre-trained stable diffusion on few-shot target datasets in an attribute-centric manner. The proposed method focuses on attribute operations, including erasure, disentanglement, regularization and enhancement.

Strengths

- The paper is well-structured and well-motivated. - The method seems sound and reasonable. - The problem to tackle is a sensible and essential problem.

Weaknesses

- It seems to me that there are several pipelines and tasks to perform in this paper, but the technical challenges should be illustrated in a concise and precise manner. It seems to me that the engineering solutions and objectives are clearly illustrates but the technical challenges are not summarized well. In this way, the technical contributions seem weak and unclear. - The proposed method seems like a combination of existing techniques. The novelty seems limited unless the authors argue specifically. - There seems to lack specific error analysis on whether the result improvement is due to the proposed method. It would be very important to demonstrate this, either quantitatively, or qualitatively. At least, in different dataset directions, e.g skeches, sunglasses. shall we demonstrate what exactly causes the improvements? Why improvements on which samples in the dataset?

Questions

Please reply with points in the weakness.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Please see weakness

Summary

This paper addresses the few-shot domain transfer problem in text-to-image diffusion generation, where the goal is to keep the style and attributes of the source examples while being able to generate images of potentially different subject matters. It binds the attributes to rarely-used tokens, but observes that these tokens could potentially induce correlated unintended features, and it adopts a prior-attribute erasure procedure to remove such correlation. It then finetunes the model for domain transfer, enforcing disentanglement of attributes and subject categories and applying attribute regularization. Additionally, for cross-category generation, it proposes an attribute enhancement technique to make the learned attributes more prominent in the generations.

Strengths

* The paper investigates an interesting task of attribute-centric transfer. * The method is well-motivated and conceptually solid. The paper observes limitations in existing approaches, such as the attribute correlation with binding tokens, and entanglement between attributes and subject categories, and effectively addresses these limitations in the proposed method. * The experiments consider a variety of settings such as intra- and inter-category transfer, and transfer with additional attribute or specific subject identity. They encompass a variety of domains and settings, and provide numerous qualitative visualizations to facilitate understanding of the method performance. The ablation studies effectively verify the contribution of each component.

Weaknesses

* The paper has discussed most of the limitations of the method, such as in cases where the examples contain multiple subject categories or multiple sub-domains, or in cases with certain local/content attributes. * One additional case is that as the method infers the attributes infers from the examples, sometimes it can pick up unintended attributes that the few-shot examples happen to share (e.g. for FFHQ glasses, cross-category generations mostly have outdoor/nature backgrounds, even though this may not be necessarily intended for this setting). It would be interesting to see extensions that allow more explicit control over including/excluding certain attributes.

Questions

* It would be helpful to have more clarifications on whether the qualitative results mostly randomly selected or hand-picked, especially given that the performance of cross-category generations are primarily demonstrated through qualitative visualizations. The paper mentions that the cross-category generations for FFHQ sunglasses in Fig 11 involved some cherry-picking, but does not seem to specify for the others. * For better understanding of the proposed method's behavior, it would interesting to explore the boundary for what attributes get encoded into the binding tokens. For example, when all examples contain one subject centered in the pictures, whether/to what extent the number, position or subject size are considered part of the attributes and get picked up by [V]. It would also be interesting to see how the method behaves when the textual prompt contains conflicting attributes with the learned binding token (e.g. "A [V] bird with blue feathers" where [V] is learned from colorless sketches).

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately discussed the limitations and societal impact.

Acknowledgement

I acknowledge and have read the author response. In the next version, I would like to see if the authors can improve the presentation and make it easier to understand the motivation and illustration of the proposed algorithm, esp the attribute erasure part. Overall, I thank for the author response and would like to keep my score.

To ACs and Reviewers: A Summary of the Discussions

At the end of the discussion period, we would like to thank you all for your time and effort. We are delighted that our DomainGallery has received many insightful comments, as well as common approval from all the reviewers. We believe that these discussions will definitely make our work a better one. We have already summarized the strengths of our work that recognized by the reviewers in the *Global Response to All Reviewers*. For your convenience, here we further collate the main concerns raised by the reviewers and our responses to them, and present them as below. ### Reviewer 2oCE - *Possible unexpected common attributes*: We have clarified that learning all the common attributes (either expected or unexpected) complies with the definition of domain-driven generation. We have also provided a possible solution to explicitly exclude unwanted attributes similar to what we have done to categorical attributes. - *Qualitative result selection*: We have illustrated some uncurated samples in the rebuttal PDF, demonstrating that the images generated by DomainGallery generally have good quality. - *Possible conflicting attributes*: We have shown some examples in the rebuttal PDF, where our DomainGallery has still produced some interesting results with mixed attributes. ### Reviewer Ue4P - *Technical challenges*: We have provided a summary of the four technical challenges in our work, and we will accordingly further clarify this part in the next version. - *Contribution and novelty*: We have explained that no previous work recognized and tried to settle these technical challenges, therefore our DomainGallery is indeed novel and may contribute to the whole community of generative models. - *Ablation study*: We have clarified that actually we have already conducted extensive ablation studies in Appendix B.1. ### Reviewer btQa - *Novelty of the similarity consistency loss*: We have explained that our contribution w.r.t. this part is actually the proposed strategy constructing paired source/target latent codes, so that our method is open to a variety of regularizations not limited to the similarity consistency loss we have adopted in this work. - *Comparison with subject-driven methods*: We have clarified that our goal is to mainly focus on domain-driven generation, while combining DomainGallery with personalization methods is just one of the promising directions and we prefer to leave this part to future works. - *Connection to Hybrid Domain Adaptation (HDA)*: We have discussed some key differences (incl. settings, models and generation scenarios) between HDA and our DomainGallery. ### Reviewer N3qa - *Applying DomainGallery to other subject-driven methods/T2I models*: These two are indeed promising future directions. We have respectively discussed our opinions (with some predictions on performance) towards these two kinds of potential combinations. - *Possible limitations & qualitative comparison*: We have clarified that we have actually discussed possible limitations in Appendix C. And we have also illustrated uncurated samples in the rebuttal PDF, including some less satisfying samples. Although not yet perfect, our DomainGallery generally performs well. - *Mixed domains*: We have shown some samples mixing two style domains in the rebuttal PDF, demonstrating that DomainGallery may produce some interesting results. We will accordingly improve our manuscript following the suggestions from the reviewers. Again we would like to express our gratitude to their precious comments!