Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models

Summary

This paper proposes Mix-of-Show for decentralized multi-concept customization using diffusion models. To tackle the concept conflict and identity loss problem, the authors propose the embedding-decomposed LoRA and the gradient fusion. The regionally controllable sampling is proposed to address the attribute binding and missing object problems. Experiments show that the proposed method outperforms existing methods in multi-concept customization.

Strengths

1. Most part of the paper is well-written and easy to understand. 2. Each component of the proposed method is reasonable and well-motivated. 3. The three observations are quite important and interesting to finetuning pretrained diffusion models. 4. Experiments show that the proposed Mix-of-Show outperforms existing methods in multi-concept customization.

Weaknesses

1. The gradient fusion is also adopted in CustomFusion, it is better to clarify it in the paper. 2. The biggest concern is the necessity of the decentralized design, since in current customization scenario, we can always obtain all the subjects together and we can fine-tune them all, and also the CustomFusion indicates this leads to better results. If the decentralization is not necessary, the contribution of this paper will become smaller. 3. Some minor typos, e.g., the caption for Figure 2(b) seems to be incorrect.

Questions

Please refer to the weakness part. It is better to emphasize the necessity of the decentralized design.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed the limitations in the appendix.

Summary

The paper presents Mix-of-Show, a framework that tackles the challenges of decentralized multi-concept customization in public large-scale text-to-image diffusion models. It introduces an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node, enabling limitless concept fusion. The framework also incorporates regionally controllable sampling to address attribute binding and missing object issues. Experimental results demonstrate Mix-of-Show's ability to compose multiple customized concepts, including characters, objects, and scenes, with high fidelity. Overall, the framework offers a promising solution for decentralized multi-concept customization in text-to-image diffusion models.

Strengths

The strength of Mix-of-Show lies in its comprehensive approach to decentralized multi-concept customization. It addresses the challenges of concept conflicts, identity loss, and missing objects through innovative techniques such as ED-LoRA, gradient fusion, and regionally controllable sampling. The framework enables the composition of complex and diverse concepts with high fidelity. It demonstrates promising results in generating multi-concept compositions, surpassing previous methods in terms of performance and flexibility.

Weaknesses

The paper does not extensively discuss the computational complexity and resource requirements associated with the proposed framework. While I'm not an expert in related area, the paper is generally well-written and easy to follow. At current stage, I have no further concerns regarding the technical details of the paper.

Questions

Please refer to the above weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors should demonstrate some failure cases in order to get a full grasp on the limitation of the proposed framework.

Summary

This paper focuses on the problem of multi-concept customization of diffusion model. Compared to the previous methods, this work focuses on fusing multiple customized objects into a single image with different extra conditions such as poses contours. The paper did an extensive study to analyze the roles of tuning the embedding and weights and based on this study it proposes a group of improvement to the multi-concepts customization. Based on the qualification results, the proposed methods are quite effective.

Strengths

1. Compared to previous methods, this paper gives a more flexible concept customization framework where multiple concepts and different types of conditions are supported. 2. This paper provide some insights on how embedding and model weights affects the concept finetuning differently and propose effective several modifications of the method accordingly. 3. Extensive qualification comparison is provided which proves the effectiveness of the proposed method. 4. The distributed training framework brough some insight on privacy preserving and federated learning related research.

Weaknesses

This paper only provide qualification results with limited use cases. It would be better to provide some quantification analysis on more data, following some previous concept customization works.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author has address the limitations.

Summary

This paper addresses a problem of significant practical importance, which is how to employ a number of individually fine-tuned concept plug-ins to produce a model, which can then be used to generate an image that contains any combinations of the concepts. The key method is center node fusion, which trains a fused model instead of simple arithmetic averages over tuned Lora weights.

Strengths

The paper tackles a problem of significant practical import, and it appears that the proposed method could work well. The center node fusion method is intuitive and can be implemented as is or tweaked to suit particular situations.

Weaknesses

There are many. In no particular order, - the term "decentralized" is quite misleading. What the paper meant was that the inverted embedding and weights for concept n must be trained by only knowing the data of concept n. That is not decentralized. - observations from empirical observations do not add to the paper's technical contribution; in this reviewer's opinion, they are not convincing. - fine details can influence the outcome a lot, for instance the split between the fine-tuning between embedding inversion (V is 1 token? 10 tokens?) and LoRA (dimensions).

Questions

This reviewer does not have questions to the authors.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

1 poor

Contribution

3 good

Limitations

This work is neutral in terms of potential negative societal impact.

Summary

The paper proposes a framework called Mix-of-Show for decentralized multi-concept customization of diffusion models. The framework uses embedding-decomposed LoRA for single-client tuning and gradient fusion for center-node concept fusion to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, it introduces regionally controllable sampling to extend spatially controllable sampling to address attribute binding and missing object problems in multi-concept sampling.

Strengths

1. The paper proposes a framework called Mix-of-Show for decentralized multi-concept customization of diffusion models. It addresses the challenges of concept conflicts and identity loss during model fusion, and demonstrates the capability of composing multiple customized concepts with high fidelity. 2. The logical flow of the paper makes it easy for readers to understand and follow the research process. 3. The paper provides detailed analyses of the experimental results.

Weaknesses

1. In Figure2(b), in the generated picture with text " V^{dogA}, a V^{cat} and a V^{dogB}, in a living room.", is the text wrong here? Or is this picture exactly the output of the model? I'm a little confused because the background of the picture is suburb rather than living room. 2. In line 230 to 231, you mentioned that Figure3(f) demonstrates that the gradient fusion preserves the concept identity, but I think the effect of this is not obvious. In other words, in the method that only uses ED-LoRA with weight fusion, the problem of identity loss basically does not exist. I can't see much difference between these two pictures. Maybe Thanos in Figure7(c) is a better example. It's a good way to find some special examples to show the effect, but I'd like to know if there is a quantitative way to accurately evaluate the effectiveness of this fusion approach. 3. In section 4, you made a qualitative comparison. Have you thought about using a quantitative approach such as CLIP score to compare your model with others, or have you ever thought about developing an new evaluation metric which can accurately tests your own approach? 4. When you use the method you proposed, have you encountered any poor-quality images? Have you done an error analysis? And have you analysed the limitations and shortcomings of these methods? 5. Some equations lacks essential explanatory notes for some of the mathematical symbols used: Equation 1, the notation of $\epsilon$ is missing 6. Some figure misses detailed explanation which causes confusion (See question 2 and 3) 7. Lack of details concerning the collected dataset and the text encoder used in the study. 8. The dataset contains different characters, objects, and scenes, however, I only see the demonstration of characters. More examples about different objects and scenes can be provided to make the results more solid. 9. There is a lack of quantitative analysis in this work. The effect of the method proposed in this paper is represented by the sample pictures, which is a qualitative analysis and can not well reflect the generalizability of the method.

Questions

Question: 1. How do you initialize V+_{class} in Line 194? Can you provide a more detailed explanation? 2. In Figure 2, what does @ mean? 3. In Figure 2, what is - [x1, y1, x2, y2]? Are they the spatial coordinates of different concepts? 4. Did you consider conducting human evaluation to make the results more convincing? Suggestions: 1. The related work content of the paper is less. It would be beneficial to expand a more detailed explanation of the relevant content. 2. A visual explanation of the overall architecture that includes the individual modules is useful, as it will help the readers to better understand the structure of the new framework. 3. The experiment section needs to introduce more details about the dataset, such as data quantity, source, input. This would provide readers with a better understanding of experiment details.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes

Further feedback about evaluation results.

Dear Reviewer B3ui, **We have included the quantitative evaluation and analysis of our Mix-of-Show in the supplementary materials and the attached PDF in our global response.** The quantitative analysis of our main components, ED-LoRA and gradient fusion, is listed in Table 1 within the attached PDF of the global response. Additionally, the quantitative evaluation for comparison with previous methods (Custom Diffusion[1], LoRA[2], and P+[3]) is presented in both Table 1 and Table 2 in the supplementary materials. For your convenience and reference, we list a portion of the important results as follows. For the comprehensive quantitative evaluation, we kindly refer you to the submitted global response PDF and our supplementary materials. --- **1) Quantitative analysis of the effectiveness in reducing identity loss through ED-LoRA and Gradient Fusion.** **CLIP Image Alignment Evaluation**: We evaluate the CLIP image alignment of the sampling results **before and after concept fusion**. This evaluation allows us to quantify the identity loss that occurs during multi-concept fusion. As shown in the table below, our Mix-of-Show (ED-LoRA + Gradient Fusion) achieves the lowest identity loss during multi-concept fusion. | Methods | Real-Objects (Single→Fused) | Real-Characters (Single→Fused) | Real-Scenes (Single→Fused) | Mean Change | | ---- | ---- | ---- | ---- | ---- | | LoRA + Weight Fusion | 0.864→0.778 (-0.086) | 0.761→0.555(-0.206) | 0.824→0.769 (-0.055) | 0.816→0.701 (-0.115) | | ED-LoRA + Weight Fusion | 0.868→0.798 (-0.070) | 0.802→0.634 (-0.168) | 0.858→0.816 (-0.042) | 0.843→0.749 (-0.094) | | ED-LoRA + Gradient Fusions | 0.868→0.846 (-0.022) | 0.802→0.770 (-0.032) | 0.858→0.838 (-0.020) | **0.843→0.818 (-0.025)** | --- **Human Preference Study**: We utilize Amazon Mechanical Turk to conduct a human preference study on weight fusion and our gradient fusion for fusing ED-LoRAs. As demonstrated in the table below, users prefer our gradient fusion over weight fusion for multi-concept fusion. | Methods | Image Alignment | Text Alignment | | ---- | ---- | ---- | | ED-LoRA + Weight Fusion | 33.5% | 47.5% | | ED-LoRA + Gradient Fusions | **66.5%** | **52.5%** | --- **2) Quantitative benchmark to previous methods.** We adopt CLIP image alignment following Custom Diffusion [1] to make a quantitative comparison with previous methods. In comparison to the embedding-only tuning approach (P+ [3]), our Mix-of-Show method achieves **much better image alignment**. When compared to Custom Diffusion [1] and LoRA [2], our method demonstrates **minimal identity loss**, resulting in **much better image alignment after multi-concept fusion**. | Methods | Real-Objects (Single→Fused) | Real-Characters (Single→Fused) | Real-Scenes (Single→Fused) | Mean Change | | ---- | ---- | ---- | ---- | ---- | | P+ | 0.790→0.790 (-) | 0.670→0.670 (-) | 0.796→0.796 (-) | 0.752→0.752 (-) | | Custom Diffusion | 0.842→0.808 (-0.034) | 0.714→0.694 (-0.020) | 0.804→0.750 (-0.054) | 0.787→0.751 (-0.036) | | LoRA | 0.864→0.778 (-0.086) | 0.761→0.555(-0.206) | 0.824→0.769 (-0.055) | 0.816→0.701 (-0.115) | | Mix-of-Show (Ours) | 0.868→0.846 (-0.022) | 0.802→0.770 (-0.032) | 0.858→0.838 (-0.020) | **0.843→0.818 (-0.025)** | --- **Reference**: [1] Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023. [2] LoRA: https://github.com/kohya-ss/sd-scripts [3] P+: Extended Textual Conditioning in Text-to-Image Generation. Arxiv 2023.

With regard to the term decentralized

In this reviewer's opinion, the term "decentralized" refers to having as little centralized control as possible, optionally compared to a commonly known centralized version. In the simple scenarios addressed in this submission, the Lora for concept n is trained entirely without any concern for the other nodes. I agree that this is de-centralized. However, this "train a lora for a concept" is also commonly done in a centralized training scenario, there is nothing that says a centralized node must mix the data of the n concepts to train one single Lora. So the contrast between centralized and de-centralized is very limited. Hence my remark that the use of the term is misleading. For the followoing, we can agree to disagree on the definition fo the term "decentralized", as I can be considered nitpicking. Federated learning, as when the term was coined, is a large number of distributed computations of gradients that are tightly controlled by a central node (parameter server). Furthermore, before launching the training, in practice a central node does careful leaf node qualification / balancing to ensure that good data enter federated training. As such, FedAvg is considered centralized training, not decentralized, by this reviewer.

Further Response To Reviewer acdR about Quantitative Analysis.

Dear Reviewer acdR, We have outlined our quantitative results in the rebuttal. The quantitative analysis of our main components, ED-LoRA and gradient fusion, is listed in Table 1 within the attached PDF of the global response. Additionally, the quantitative evaluation for comparison with previous methods (Custom Diffusion[1], LoRA[2], and P+[3]) is presented in both Table 1 and Table 2 in the supplementary materials. Here, we post the key results for your convinence. For the comprehensive quantitative evaluation, we kindly refer you to the submitted global response PDF and our supplementary materials. Hope those results can address your concerns. --- **1) Quantitative analysis of the effectiveness in reducing identity loss through ED-LoRA and Gradient Fusion.** **CLIP Image Alignment Evaluation**: We evaluate the CLIP image alignment of the sampling results **before and after concept fusion**. This evaluation allows us to quantify the identity loss that occurs during multi-concept fusion. As shown in the table below, our Mix-of-Show (ED-LoRA + Gradient Fusion) achieves the lowest identity loss during multi-concept fusion. | Methods | Real-Objects (Single→Fused) | Real-Characters (Single→Fused) | Real-Scenes (Single→Fused) | Mean Change | | ---- | ---- | ---- | ---- | ---- | | LoRA + Weight Fusion | 0.864→0.778 (-0.086) | 0.761→0.555(-0.206) | 0.824→0.769 (-0.055) | 0.816→0.701 (-0.115) | | ED-LoRA + Weight Fusion | 0.868→0.798 (-0.070) | 0.802→0.634 (-0.168) | 0.858→0.816 (-0.042) | 0.843→0.749 (-0.094) | | ED-LoRA + Gradient Fusions | 0.868→0.846 (-0.022) | 0.802→0.770 (-0.032) | 0.858→0.838 (-0.020) | **0.843→0.818 (-0.025)** | --- **Human Preference Study**: We utilize Amazon Mechanical Turk to conduct a human preference study on weight fusion and our gradient fusion for fusing ED-LoRAs. As demonstrated in the table below, users prefer our gradient fusion over weight fusion for multi-concept fusion. | Methods | Image Alignment | Text Alignment | | ---- | ---- | ---- | | ED-LoRA + Weight Fusion | 33.5% | 47.5% | | ED-LoRA + Gradient Fusions | **66.5%** | **52.5%** | --- **2) Quantitative benchmark to previous methods.** We adopt CLIP image alignment following Custom Diffusion [1] to make a quantitative comparison with previous methods. In comparison to the embedding-only tuning approach (P+ [3]), our Mix-of-Show method achieves **much better image alignment**. When compared to Custom Diffusion [1] and LoRA [2], our method demonstrates **minimal identity loss**, resulting in **much better image alignment after multi-concept fusion**. | Methods | Real-Objects (Single→Fused) | Real-Characters (Single→Fused) | Real-Scenes (Single→Fused) | Mean Change | | ---- | ---- | ---- | ---- | ---- | | P+ | 0.790→0.790 (-) | 0.670→0.670 (-) | 0.796→0.796 (-) | 0.752→0.752 (-) | | Custom Diffusion | 0.842→0.808 (-0.034) | 0.714→0.694 (-0.020) | 0.804→0.750 (-0.054) | 0.787→0.751 (-0.036) | | LoRA | 0.864→0.778 (-0.086) | 0.761→0.555(-0.206) | 0.824→0.769 (-0.055) | 0.816→0.701 (-0.115) | | Mix-of-Show (Ours) | 0.868→0.846 (-0.022) | 0.802→0.770 (-0.032) | 0.858→0.838 (-0.020) | **0.843→0.818 (-0.025)** | --- **Reference**: [1] Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023. [2] LoRA: https://github.com/kohya-ss/sd-scripts [3] P+: Extended Textual Conditioning in Text-to-Image Generation. Arxiv 2023.

About the new results

The new results have convinced me to some extent, but compared to Custom Diffusion and others, the improvement is not significant. However, considering the novelty of this method, I have raised my score.