Summary
This paper proposes a T2I personalization approach based on low-rank residuals, which uses fewer parameters and does not rely on class regularization. A localized attention-guided sampling approach is proposed such that the learned concept only influences (the subject) parts of the image, preventing overfitting the learned concepts while generating subject-irrelevant content (like background).
Strengths
The paper leverages LoRA to learn personalized concepts, despite being widely used by the community [1, 2], and discovers that such usages can preserve the existing diffusion prior, thus eliminating the need for class regularization. The paper also experiments with the optimal values of the LoRA rank, which may be useful to the community.
Weaknesses
My primary concerns about the papers are two-fold.
1. Method-wise, the main contributions of this paper are to use LoRA for fewer trainable parameters and getting rid of regularization, and use attention maps to mask out the subject foreground. However, as mentioned in the strength part, using LoRA for training diffusion models is widely adopted in the communities [1], as well as for training personalized diffusion models [2]. Despite the paper exploring that leveraging LoRA preserves diffusion prior thus avoiding regularization, the reduction in trainable parameters is mainly contributed by the use of LoRA, thus the contribution is very incremental in my opinion. Secondly, previous works have also adopted cross-attention maps to guide the diffusion model to focus on the subject token such that the attention maps associated with each attended token will be activated [3]. More similarly, [4] leverages attention maps to suggest personalized models for foreground subject generation, which shares the same motivation and almost identical solution as this paper, while according to the time stamp it was released around half a year ago with their code available (June 1st, 2023 according to https://arxiv.org/abs/2306.00971).
2. Performance-wise, it is hard to distinguish the superiority of the proposed approach against DreamBooth. The comparison might be unfair qualitatively, however, as shown in Table 1, the proposed method does not lead to a compelling improvement against baselines like DreamBooth on CLIP text and image alignment metrics. Furthermore, as suggested in Table 2, more human evaluations indicate preferences over DreamBooth for better text alignment as well.
As above, from both the perspectives of presented methods and experimental evaluations, the paper does not show a clear distinction between the existing approaches.
[1] https://github.com/cloneofsimo/lora
[2] https://github.com/huggingface/peft/tree/main/examples/lora_dreambooth
[3] Chefer, Hila, et al. "Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.
[4] Hao, Shaozhe, et al. "ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation." arXiv preprint arXiv:2306.00971 (2023).
Questions
Please see the weaknesses above.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.