MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts

Summary

This paper aims to address the quality issues in human-centric text-to-image generation by constructing a large-scale dataset of millions of portraits. Additionally, the authors propose training experts for generating facial and hand details and efficiently integrate these experts using the MoE architecture into diffusion models. This integration enhances the realism of the final outputs in terms of facial and hand details.Extensive visualizations and analyses demonstrate the effectiveness of the proposed approach in this paper.

Strengths

+This paper is well-written, clearly expressing the motivation and key contributions. +The proposed Mixture of Low-rank Experts on text-to-image diffusion model framework is novel. To the best of my knowledge, this is the first work to try MoE learning strategy to enhance the human generation quality in diffusion models. +The contributions are relatively substantial, especially with the author's collection of a publicly available high-quality human dataset. This dataset's availability will advance the community in deeper research into human generation. +The visualized comparisons are comprehensive and demonstrate the advantages of the MoE architecture in diffusion models.

Weaknesses

Overall, I am quite satisfied with this paper, but I still have some questions or concerns, as detailed below: -- The text description process is inadequate, particularly regarding the manual filtering of data generated by the LLaVA model, which lacks any detailed description. As far as I know, LLaVA is trained on general image-text pairs, so there should be many hallucinated descriptions in the process of generating human image descriptions. How did the authors address this issue? Additionally, why did the authors not consider using some human/clothing attribute prediction methods and then use the generated attributes as prompts for the MLLM to make the results more reliable? --The visualized comparison results presented by the authors are mostly half-body portraits. Could the authors show some full-body comparison results? In these images, the face and hand areas are smaller, increasing the probability of poor quality and making them more challenging to fix

Questions

Most of my questions have been listed in the "weaknesses" section. Please respond to the above questions. Additionally, to better evaluate its generalization capability, have the authors considered applying the MoE architecture to transformer-based diffusion models, such as PixArt[1]? [1] Junsong Chen, Jincheng Yu and etc. PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR, 2024.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

See the "weakness" and "questions" section.

Summary

This paper enhances these Text-to-image diffusion models by introducing a curated dataset of over a million human-centric images and a novel method, MoLE, which utilizes specialized low-rank modules to improve facial and hand image quality in diffusion processes. If the rebuilt dataset can be race-balanced and the approach could promote racial balance, I will be glad to raise my score.

Strengths

1. The paper proposes a new human-centric dataset to enhance the human-centric generations, which is interesting. 2. Extensive experiments demonstrate the effectiveness of this approach. 3. The presentation is clear and easy to follow.

Weaknesses

1. Stable Diffusion has introduced race biases when generating images. MoLE focuses on the generation qualities of faces and hands. If MoLE is beneficial to alleviate race biases with the reconstruction of the human-centric dataset, it will be more impactful like [1]. 2. Limb deformation is also a big question in Stable Diffusion[2][3]. With the introduction of many human images featuring limbs in this dataset, it is crucial to assess whether this approach can effectively mitigate these deformations. 3. The generation of two global scalars may be tricky (Lines 180-182), current experiments show less evidence about the effects of the global scalars. [1] ITI-GEN: Inclusive Text-to-Image Generation [2] HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation. [3] Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation.

Questions

4. From lines 88-89, the main race is the white. Could the authors provide an analysis of the ratios of different races in one prompt (A Beautiful Woman)? Also, could the authors provide a discussion that if the ratios of different races are the same, will the generations of different races be improved? 5. The values of S in Figure 1 are more arbitrary to decide the qualities of a generation, so how to decide these values in MoLE? 6. Since the dataset is considered to be part of the innovation, will the dataset be released in the future?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The negative board impacts are insufficient in the Conclusion. The authors should analyze whether their approach will introduce more biases on race and other impacts, such as fake faces.

Summary

This paper aims to explore human-centric text-to-image generation, particularly in the context of faces and hands, the results often fall short of naturalness due to insufficient training priors. The authors alleviate the issue from two perspectives. 1) The authors collect a human-centric dataset with two specific sets of close-up images of faces and hands. 2) The authors propose Mixture of Low-rank Experts (MoLE) method by considering low-rank modules trained on close-up hand and face images respectively as experts.

Strengths

1)This paper constructs a human-centric dataset comprising over one million high-quality human-in-the-scene images and two specific sets of close-up images of faces and hands. These datasets collectively provide a rich prior knowledge base to enhance the human-centric image generation capabilities of the diffusion model. 2)This paper proposes a simple yet effective method called Mixture of Low-rank Experts (MoLE) by considering low-rank modules trained on close-up hand and face images respectively as experts. 3)The paper is well-written and easy to follow. The dataset and benchmark of this paper are open source.

Weaknesses

1) As the author mentioned, the soft mechanism is built on the fact that each token can adaptively determine how much (weight) should be sent to each expert by the sigmoid function. So, it will be better if the author can provide and analyze the distribution of the weight sent to each expert by the sigmoid function. 2) There are some minor writing errors in the paper, such as "is is" on line 208. The author needs to carefully check the manuscript.

Questions

Please check the weakness.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of the proposed dataset and method.

Summary

The authors propose a large-scale dataset for human image generation, comprising over one million images, including close-up face and hand subsets. They also introduce MoLE (Mixture of Low-rank Experts), a novel framework that utilizes two low-rank experts to learn face and hand representations. This approach shows promise for generating high-quality human images with precise control over face and hand features. Overall, the proposed dataset and method are a valuable contribution to the field of human image generation.

Strengths

1. A large-scale human-centric dataset is proposed along with two close-up face and hand subsets. 2. A MoLE framework with two experts is proposed which is novel and interesting.

Weaknesses

1. Comparison with existing datasets: While the proposed dataset is a significant contribution, a thorough comparison with established datasets like CosmicMan is essential to contextualize its value. 2. Comparison with state-of-the-art methods: The paper primarily focuses on generating realistic faces and hands, but it lacks a comprehensive comparison with relevant methods like HanDiffuser and HyperHuman. Comparing only with Stable Diffusion (SD) may not provide a complete picture. It would be beneficial to train existing methods on the proposed dataset and compare the results with the proposed method. 3. Ablation study on experts: It would be interesting to see the results of training only one expert compared to training two experts, to understand the impact of using multiple experts on the model's performance.

Questions

My main concern is the comparison with existing datasets and methods. Could you provide more details about it?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Response to Ethics Reviewer oo1E

We thank the reviewer for these valuable comments! Below is our pointwise response. Thank the reviewer for the time and effort. **Recommendation 1:** Dataset Diversity: Future work should aim to enhance the diversity of the dataset to better represent different demographics and real-world scenarios. **Answer 1:** We thank the reviewer for the suggestion. Similar to the famous FFHQ dataset [1], our dataset inevitably inherits the potential bias of target websites. However, we agree that it will be more meaningful for our dataset to represent different demographics and real-world scenarios better. We are actively working on collecting more race-balanced data and are working hard to further alleviate this issue. In the future, we will enhance the diversity of our dataset. We will add a future work section in our paper to highlight this discussion. **Recommendation 2:** Explicit Consent: When possible, obtaining explicit consent from individuals whose images are used in the dataset can further mitigate privacy concerns. **Answer 2:** We thank the reviewer for the advice. Although the images in our dataset are collected from websites that publish content under the Public Domain CC0 1.0 license—already allowing for free use, redistribution, and adaptation for non-commercial purposes—we are willing to make every effort to obtain explicit consent from the individuals whose images are used in our dataset to further mitigate privacy concerns. **Recommendation 3:** Ongoing Ethical Review: Implementing an ongoing ethical review process to monitor the use and potential misuse of the generated images and the dataset. **Answer 3:** We thank the reviewer for the advice. We primarily consider implementing an ongoing ethical review process from the following four aspects: 1. Transparent Usage Policy. We develop and publicly publish a clear usage policy for the dataset and generated images, outlining permitted and prohibited uses. All users are required to sign an agreement, committing to adhere to these policies. 2. User Qualification Assessment: For the application for the usage of generated images and the dataset, we carefully review the applicant's qualifications, purpose of use, possible risks, etc. Additionally, we will strictly control access to the dataset, allowing only authorized personnel to interact with the data. 3. Regular Review of Usage: We regularly review how the applicant used our dataset and generated images and whether the usage complies with the original consent scope and assess the risk of data misuse or unauthorized disclosure. 4. User Feedback Mechanism. We will establish a feedback mechanism that allows users to report any ethical concerns or issues they encounter. We will address feedback promptly and make necessary adjustments to our policies or practices. **Recommendation 4:** Public Awareness and Education: Educating the public about the capabilities and limitations of such technology can help mitigate misuse and ensure that it is used ethically. **Answer 4:** We thank the reviewer for the advice and agree that educating the public about the capabilities and limitations of such technology can help mitigate misuse. Therefore, we will revise and add a dedicated section that explicitly presents the potential limitations and ethical considerations of our approach. By doing so, we aim to ensure that the capabilities of our method are not overstated and its limitations are clearly understood. Furthermore, we will include practical examples to illustrate scenarios where misuse could occur and emphasize the importance of adhering to ethical guidelines when applying our method. We believe that these additions will contribute to a more informed and responsible use of the technology, aligning with the broader goal of promoting ethical practices in our field. We hope that our responses above can address your concerns. Thanks for the time and effort again. [1] A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR2019

Response to Ethics Reviewer xqKQ

Thank the reviewer for the feedback. We appreciate the thorough review and are glad to hear that no concerns were identified regarding the ethical aspects of the work. Thank the reviewer for the time and effort.