Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models

Summary

MuDI is a novel framework designed for multi-subject personalization in text-to-image diffusion models. It effectively decouples identities of multiple subjects, using segmented subjects from a foundation model for data augmentation and generation initialization. A new metric is introduced to evaluate multi-subject personalization. Experimental results show MuDI produces high-quality personalized images without identity mixing, outperforming existing methods.

Strengths

- This paper is well-written and easy to follow. - The experimental results are comprehensive and sufficiently support the claims made in the paper. - MuDI successfully prevents identity mixing when generating multiple subjects, maintaining clear individual identities. - Introduction of a new metric provides a better assessment of multi-subject personalization performance.

Weaknesses

- Inheriting from DreamBooth, MuDI requires test-time fine-tuning for each set of concepts, which might hinder its practical application. In contrast, some approaches like IP-Adapter can achieve customization in a tuning-free manner. - There is a spelling error on line 796. Additionally, there is a miscitation on line 690, where FastComposer[51] is incorrectly referred to as a "single-subject personalization" method; it is actually a zero-shot multi-subject customization method.

Questions

- During training, MuDI sets the background to white, which might reduce text alignment in the generated images. However, in Figure 6, MuDI does not seem to be affected by this. I need the authors to provide a possible explanation for this discrepancy. - The proposed strategy of modifying the initial noise during inference appears to be very effective. I am curious whether DreamBooth with Region Control, if applied with the same inference strategy, would achieve better performance than MuDI.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

No negative societal impact.

Summary

This work introduces a training and inference pipeline and an evaluation benchmark for multi-concept customization for text-to-image diffusion models. Specifically, the pipeline comprises a Seg-Mix training stage, which can be viewed as a data augmentation trick to prevent the fine-tuned model from learning mixed attributes, and a mean-shifted noise initialization for the very first step of the denoising process, aiming to inject appearance and layout priors. The evaluation benchmark takes into consideration the dysfunctionality of previous methods when evaluating the disentanglement extent between customized subjects. The benchmark utilizes a Detect-and-Compare workflow considering the similarity for the same subject and dissimilarity between two distinct subjects.

Strengths

1. The paper is well-written and easy to follow. I didn’t encounter many confusing expressions during my first reading. 2. The introduced methods are intuitively effective, and the design of the evaluation benchmark is convincing for evaluating the extent of disentanglement. 3. This work introduces two interesting additional uses: size control and modular customization, which are helpful for improving the application scenarios for customization.

Weaknesses

1. Though the method is intuitively effective, as I stated in Strength #2, the novelty is limited since some designs have been experimentally verified by previous methods, such as the utilization of descriptive class [1] and a data-augmentation pipeline for multi-concept customization [2]. 2. Though the proposed method seems reasonable for combining distinct subjects with different region locations, like a cat and a dog sitting beside each other, it may be challenged in scenarios where the subjects have rich semantic interactions, like a person wearing glasses. This limitation is induced by the region-control designs, where both augmented training and the initialization during inference pose strong regularization for decoupling two instances. This limitation makes this method difficult for general multi-concept customization. [1] InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetunin [2] SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Questions

Seg-Mix uses a white background and a prompt organized as “…, simple background”, my question is that is this design posing the risk of outputting more images with white backgrounds even when using edited prompts? This seems inevitable, much like how Cut-Mix inherits the stitching artifacts from the training data.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

See Weakness#2

Summary

The paper introduces MuDI, a novel framework designed to improve multi-subject personalization in text-to-image diffusion models. Unlike current methods that often mix identities and attributes from different subjects when generating multiple subjects simultaneously, MuDI effectively decouples these identities. The framework employs segmented subjects generated by a foundation model for segmentation, known as Segment Anything, which is used for both training and inference. This approach serves as data augmentation for training and as initialization for the generation process. Additionally, the authors introduce a new metric to better evaluate the performance of their method in multi-subject personalization.

Strengths

1. The topic is interesting and it has a good novelty. 2. The presentation is good and the results look promising.

Weaknesses

The dataset is small, and more analysis should be made. Please see the detailed Questions.

Questions

1. The dataset provided in the paper is small and monolithic in style, which does not adequately illustrate the superiority of the method. Could the authors conduct more experiments to clarify this concern? 2. I'm concerned about whether image content, including resolution, style, and other factors impacts performance. Could the authors clarify this point? 3. Relying solely on the proposed metric could not demonstrate the model's superiority. Could the authors include comparisons with other existing metrics to further validate the model's performance? 4. The qualitative comparison in Section 5.2 lacks comprehensiveness. Displaying a few examples is not enough to ensure credibility. It is recommended to include quantitative comparisons, such as the proportion of high-quality generated results and other quantifiable metrics. 5. The paper lacks detailed explanations of the important evaluation metrics for multi-subject fidelity and text fidelity, making it difficult for readers to understand their specific practical significance and thus grasp the model's advantages. It would be helpful if the authors could provide more detailed explanations.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

The paper proposes MuDI, a novel method for generating images with multiple personalized subjects. By leveraging segmented subjects from reference images for both training and inference, MuDI effectively addresses the challenge of identity mixing in multi-subject image generation. Key contributions include a new data augmentation technique (Seg-Mix) and a new evaluation metric for multi-subject fidelity.

Strengths

- The paper introduces a novel approach to multi-subject image generation by leveraging segmented subjects for both training and inference, effectively decoupling subject identities. This represents a creative combination of existing techniques in image segmentation and text-to-image generation. - The paper is well-structured and clearly presented, with a solid experimental evaluation demonstrating the effectiveness of the proposed method. - The authors effectively communicate the problem, the proposed solution, and the experimental results. The paper is well-organized and easy to follow. - By addressing the critical challenge of identity mixing in multi-subject image generation, the paper offers a valuable contribution to the field. The proposed method has the potential to significantly impact applications requiring the generation of multiple distinct subjects within a single image.

Weaknesses

- While the paper presents a comparative analysis with existing methods, a more comprehensive evaluation against a wider range of baselines, including recent advancements in image generation and personalization, would strengthen the paper's claims. It would be essential to compare with methods like PortraitBooth (CVPR 2024) and FastComposer. - Additionally, exploring different evaluation metrics beyond the proposed D&C metric could provide a more holistic assessment of the method's performance. - The paper lacks sufficient details about the dataset used for training and evaluation. A more in-depth description of the dataset, including its size and diversity, would enhance the reproducibility of the work. - Although the paper includes some ablation studies, a more comprehensive analysis of the impact of different components of the proposed method (e.g., Seg-Mix, initialization, descriptive class) on the overall performance would provide deeper insights into the method's effectiveness. - While the paper acknowledges the limitations of existing methods, a more thorough discussion of the potential limitations of the proposed MuDI method, such as its sensitivity to image complexity or its performance on highly similar subjects, would strengthen the paper's overall contribution. Some studies on how the size of the objects composed using SegMix during training affects the model, i.e, does it lead to any size biases in the model?

Questions

- Could the authors provide more details about the dataset used for training and evaluation, including its size, diversity, and collection process? Additionally, a more in-depth description of the evaluation metrics and experimental setup would enhance reproducibility. - How does MuDI compare to other state-of-the-art methods that focus on image composition or layout control for multi-subject image generation? Specifically for PotraitBooth and FastComposer?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

- Could the authors elaborate on the limitations of MuDI, such as its performance on highly similar subjects or complex scenes?