AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation

Summary

This paper proposes a new method for personalized image generation, decompose the personalization process into three training stages and introducing a cross-attention map regularization term.

Strengths

The manuscript is well-written. The authors propose to address the intrinsic issues of two classical and up-to-date methods, Textual Inversion and DreamBooth, by aligning the new concept from the prompt while preserving the original subject identity. This approach tackles a significant problem.

Weaknesses

1. Some parts of the manuscript are slightly verbose. For instance, the introduction section already introduces the existing problems of Textual Inversion and DreamBooth, along with their corresponding analysis and naïve solution. However, these points are reiterated in Section 4.1 and the first paragraph of Section 4.2 without adding any new information. 2. What does the term 'this approach' refer to in Line 174? 3. In general, the proposed method tends to be a bag of tricks with some customized hype-parameters. 4. Important comparisons are lacking in the manuscript, specifically the comparison with Textual Inversion (TI) and DreamBooth (DB) individually, in addition to the comparison with the combined approach of TI+DB. 5. This method uses cross-attention map for regularization, which results in a high time cost; on a NVIDIA A100, training requires 660 steps, taking 20 minutes. In contrast, DreamBooth requires only 5 minutes to train for 1000 steps.

Questions

1. Do the constraints imposed between [V] and [super-category] across all layers in Eq. 2 potentially restrict the diversity? Considering that [super-category] represents the original identity information and [V] introduces diversity. 2. Why was SD2.1 chosen, and how do other models perform? 3. In line 197, it is mentioned that retaining the prior preservation loss leads to poor results. Can you provide experimental results to support this?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper proposes a method to enhance the performance of personalizing text-to-image models by appropriately combining textual inversion approach, which learns new embeddings, and DreamBooth approach, which fine-tunes model weights. They demonstrate the effectiveness of their approach through qualitative evaluation and human evaluation.

Strengths

* Research on appropriately combining textual inversion and DreamBooth methods has been needed in this field, and they propose a reasonable and clear method for this. * The experimental results are good.

Weaknesses

* (minor) The proposed method seems quite similar to the Magicapture [1] approach in that it separates embedding learning and weight fine-tuning and conducts regularization on attention. However, this paper does not cover a comparison with Magicapture. * (minor) The ablation study is not conducted quantitatively and relies on a single generation scenario. [1] Hyung, Junha, Jaeyo Shin, and Jaegul Choo. "Magicapture: High-resolution multi-concept portrait customization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

Questions

When conducting analysis in Figure 1, how many steps was DreamBooth trained for to obtain the results? It seems that with sufficient training steps, DreamBooth might exhibit different results.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The author proposed a method to generate high-quality personalized images. First, a textual embedding is learned, then the cross-attention layers is finetund to refine attention map during learning the textual embedding, finally, the entire U-Net is trained to capture the subject identity.

Strengths

1. The paper is welll-written and easy to understand. 2. The proposed method achieve competitive results comparing with SOTAs. 3. Using the super-category attention map is an interesting idea to calibrate the [V] attention map in a self-supervised manner.

Weaknesses

1. The proposed method needs a costly test-time optimization to generate personalized images. The 3 stage finetuning requires expensive computation (20 minutes in A100) 2. Even though the Attention Map Regularization idea is interesting, it seems to be the only critical contribution in this paper. The authors combine some techniques from off-the-shelf method including CostumDiffusion/DB/TI, and conducted them step by step with proposed -attention map regularization.

Questions

1. Where does the super-category label come from? Is it a part of the annotation? 2. I am curious how the TI+DB perform if it is optimized step by step like AttnDreamBooth, e.g. first tuning the textual embedding and then the U-Net or first tuning U-Net then optimizing the embedding.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

The submission proposes AttnDreamBooth for text-to-image personalization. It addresses the limitations of existing methods, Textual Inversion and DreamBooth, by separating the learning process into three stages: embedding alignment, attention map refinement, and subject identity capture. The method aims to improve identity preservation and text alignment in generated images. # Key Contributions: - Introduces an approach for text-to-image personalization. - Addresses the limitations of existing methods by separating the learning process. - Demonstrates improved performance in terms of identity preservation and text alignment. - Provides a comprehensive analysis of the proposed method through qualitative and quantitative evaluations.

Strengths

-The paper identifies a key challenge in text-to-image personalization and provides a novel solution. - The proposed method, AttnDreamBooth, demonstrates superior performance in both identity preservation and text alignment compared to Textual Inversion and Dreambooth. - The work contributes to advancing the field of text-to-image personalization by offering a more effective approach to balancing the trade-off between identity and text alignment.

Weaknesses

- My main concern is comparison with new existing work. There have been several recent works after Textual inversion and Dreambooth that have significantly advanced this area. Some of them like SuTI (NeurIPS 2023), Instruct Imagen (CVPR 2024), HyperDreambooth (CVPR 2024) are also zero-shot which do not require any fine-tuning during evaluation. - It is critical for this work to compare with the state-of-the-art papers in this area. I have listed a few above. However, I am sure that there could be more papers in the past two years in this area.

Questions

- Comparison with multiple recent baselines is required. It would be great if the authors can provide this.

Rating

3

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

- While the proposed approach is novel, the core idea of decomposing the personalization process into multiple stages is not entirely groundbreaking. A more comprehensive exploration of the relationship between the proposed method and existing work would be beneficial.