PuLID: Pure and Lightning ID Customization via Contrastive Alignment

Summary

In this paper, an ID customization method for text-to-image generation without tuning is studied. In practical applications, the method often encounters the problems of lack of ID fidelity and interference of the original pattern line by inserting ID. To solve this problem, the author proposes a new ID customization method for text-to-image generation without tuning. A large number of experiments, called Pure and Lightning ID customization (PuLID), on multiple benchmark datasets validate the superiority of the proposed PuLID.

Strengths

The paper is well motivated, cleverly designed and clearly illustrated. The experimental results are obviously superior to other methods.

Weaknesses

The author's work is very good, but I still have three questions I would like to discuss with them. 1. Experimental results show that PuLID has achieved good performance on specific data sets, but the generalization ability of the model on different types and styles of data sets has not been discussed much in this paper. Can the author provide more experimental results on the generalization of the model? In addition, is there further analysis and discussion of possible limitations of the model, such as its performance in handling certain ID features or style transitions? 2. The authors mentioned that in order to solve the two problems mentioned in their method, the final loss calculation is based on the combination of three kinds of losses. What impact do the different proportions of the three kinds of losses have on the experimental results, especially the impact of ID loss? 3.The paper points out that high-quality images can be generated in fewer steps through the Lightning T2I branch, and ID losses can be calculated on this basis. Is there a significant improvement in computing efficiency with this method? Compared to the traditional multi-step diffusion model, what is the training and reasoning time of the Lightning T2I branch in practical applications? 4.The PULID used by the author in the experiment is based on SDXL and the 4-step SDXL-Lightning. Is it possible to conduct experiments based on other diffusion models to judge its universality？

Questions

I am mainly concerned about the universality of the proposed method and theoretical analysis, whether the proposed Lightning T2I branch can be applied to other diffusion models, and how fast and efficient it is in the actual training and reasoning process.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

In general, the content and theory of the article are very rich, but there are some limitations. 1) There are few experimental data, which is not well reflected in aspects such as universality and speed. 2) Is it possible to extend this method and replace ID with other things, such as (clothing)?

Summary

This paper introduces Pure and Lightning ID customization (PuLID). It is a tuning-free approach for customizing identities in text-to-image generation. The model is trained over a huge dataset. Technically, it integrates a Lightning T2I branch alongside a standard diffusion model, incorporating contrastive alignment loss and accurate ID loss to maintain fidelity to the original model. Experimental results show PuLID achieves superior performance in preserving identity fidelity and enhancing editability. Additionally, PuLID ensures consistency in image elements like background, lighting, composition, and style throughout the identity customization process.

Strengths

1. This paper is well-written as it explores the shortcomings of previous approaches in realistic scenarios, highlighting issues such as fidelity and the impact of inserting IDs into original T2I diffusion models. 2. The novelty of this paper lies in its new perspective on training paradigms with the introduction of PuLID. Unlike existing ID-preservation papers that focus on ID encoders, PuLID distinguishes itself by incorporating a Lightning T2I branch. This distinction is further enhanced through the integration of ID loss and alignment loss. 3. The experiments conducted in this study demonstrate superior performance both quantitatively and qualitatively compared to current methods.

Weaknesses

1. The primary limitation, in my opinion, is that the PuLID model is trained using an internal dataset. This raises questions about whether the observed improvements stem primarily from the introduction of this new dataset or from the efficacy of the method itself. Consequently, it's unclear whether comparisons with other methods on public datasets are fair or solely reflective of dataset differences. 2. Another picky question for the authors is to include more quantitative comparison metrics as in PhotoMaker[1] and ConsistentID[2], including DINO, Face Diversity, FID, Speed, FGIS (fine-grained identity similarity), etc. Furthermore, can you explain why the experimental results in your paper are much lower than that showed in the original PhotoMaker paper? 3. It seems the T2I lightning branch is pretty important for your method PuLID. Have you done any ablation study over the choices of the T2I lightning models, e.g. the LCM, SDXL-turbo, TCM, InstaFlow, UFOGen, etc? This is to verify your model can be based on SDXL or other backbones, and the lightning models are with diverse choices. Also you choose T=4, would it be influenced by T=1,2,8...? [1] PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding [2] ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving

Questions

Check the weaknesses

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Check the weaknesses

Summary

This article presents a novel fine-tuning-free ID customization method called PuLID for text-to-image generation tasks. The method introduces Lightning T2I and contrast alignment loss, aiming to minimize the interference with the original model behavior while maintaining high ID fidelity. Experiments show that PuLID achieves excellent performance in terms of both ID fidelity and editability.

Strengths

1. Introducing Lightning T2I branching and contrast learning approaches to achieve better performing ID customization methods. 2. The authors provide a wealth of ablation experiments to demonstrate the effectiveness of the method.

Weaknesses

Here are some points and suggestions on how the authors should address them: Clarification of Contributions: The authors should clearly delineate their contributions in the manuscript. If the primary contribution is the use of contrastive alignment loss and ID loss, they should emphasize how these methods specifically address the problem of high ID similarity and reduce ID interference. They should provide a more detailed explanation of how these losses are implemented and why they are effective. Related Work: The authors should expand their discussion on related work to include whether contrastive learning has been applied in similar contexts. This would help situate their work within the broader field and demonstrate its novelty. Method Section: There should be a more detailed explanation of how the method was realized. This includes a clear description of the steps taken, the rationale behind the chosen approach, and any unique aspects of the implementation. Theoretical Analysis: The authors should provide a more in-depth theoretical analysis, particularly for components like L_align-sem. They should explain the theoretical underpinnings of their approach and how it contributes to the overall effectiveness of the method. Discussion on Main Contributions: The manuscript should include a richer discussion on the main contributions. For example, the authors should explore whether the proposed loss functions reduce the weight of ID in text-to-image generation tasks. They should provide evidence or reasoning to support their claims. Ablation Studies: The authors should conduct ablation experiments to validate the necessity and effectiveness of each component of their method. For instance, if L_align-layout is introduced without ablation studies, it's unclear how crucial this component is to the overall performance. Experimental Results: While the effectiveness is illustrated in the experimental results, the authors should ensure that these results are robust and that the experiments are designed to test the method under various conditions. Layout Consistency: The authors should address the issue of layout consistency more thoroughly. If L_align-layout is mentioned but not rigorously tested, they should either conduct additional experiments or provide a theoretical justification for its inclusion. Overall Clarity and Depth: The authors should ensure that the manuscript is clear and that the depth of the discussion matches the complexity of the problem. This includes explaining the limitations of their approach and how it compares to existing methods. By addressing these points, the authors can provide a more comprehensive and convincing argument for their contributions, making their research more accessible and impactful to the academic community.

Questions

1. Lines 133 and 195 both mention that Q is an image feature of UNet, could the authors please explain how that image feature was obtained. 2. Is L_align-layout necessary? Is there any connection to your solution to the problem of ID information polluting the prompt? 3. In addition I would like to know how the authors' approach relates to the CFG scale (Classifier-Free Guidance scale) and whether CFG is valid for this task.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

YES

Summary

The paper propose a tuning-free method for customization text-to-image diffusion model. Particularly, the author propose to utilize efficient diffusion model (SDXL-lightning in this case) to generate samples during training. Then, they adopt contrastive alignment loss to preserve the identity of subject in the input.

Strengths

- The paper is fairly well-written and easy to read. - The proposed method is well-motivated and sound. - The ablation experiment demonstrates the effectiveness of each proposed components. - Reported results seem quite impressive.

Weaknesses

- Since the model requires an efficient text-to-image diffusion model (in terms of number of inference steps), it can hinder the application of the introduced method with other base models.

Questions

- Since the author roll out full diffusion path (4 steps) during training, how much memory is required for training (e.g., compared to existing methods)? - How does the model perform if we set the number of diffusion step to (1, 2, 6)? (and how much memory footprint) - Does this method work for subject other than human as input (e.g., dog, cat, etc) ?

Rating

7

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

please refer to weakness section.

Thanks for the rebuttal

My concerns are partially addressed in the rebuttal. I prefer to see this project open source and more people engage to make it function better and more effective. Here I prefer to keep my rating.