Stylus: Automatic Adapter Selection for Diffusion Models

Summary

This work presents a novel system, Stylus, designed to enhance the efficiency and effectiveness of generating high-quality images using diffusion models like Stable Diffusion. The key challenge addressed by this work is the automatic selection and composition of relevant fine-tuned adapters from a vast pool of over 100,000 available adapters, which are often poorly documented and highly customized. This work advances the field by providing a robust and automated solution by leveraging the vast number of available adapters.

Strengths

- The creation of StylusDocs, a curated dataset with 75K adapters and pre-computed embeddings, adds substantial value to provide a rich resource for further experimentation and development. - Comprehensive Evaluation: The paper provides a thorough evaluation of Stylus across multiple metrics (CLIP, FID) and diverse datasets (Microsoft COCO, PartiPrompts). This robust evaluation framework enhances the credibility of the claimed performance improvements. - Open Source and Reproducibility: By planning to release Stylus and StylusDocs as open-source resources, the authors contribute to the transparency and reproducibility of their research. This aligns well with the community’s values and encourages further developments based on their work.

Weaknesses

- Unclear Motivation for the Method: In the refiner step, the paper does not clearly explain why Gemini Ultra is trusted to generate better adapter descriptions. If other multimodal language models (MLLMs) were used, how would the results differ? - Incomplete Ablation Study: The ablation study is not comprehensive as it does not include an ablation of the refiner component. Understanding the impact of the refiner step on the overall performance of Stylus is crucial. - Quality Assurance of Adapter Descriptions: The paper does not provide sufficient details on how the quality of the adapter descriptions generated by the refiner is ensured. It is unclear whether any validation or verification steps were taken to confirm the accuracy and reliability of these descriptions. - Insufficient Description of Masking Process: The description of the masking process is not detailed enough. Specifically, the meaning of α_j in Equation 2 and the function Mask() are not adequately explained. Additionally, the masking process is not reflected in Figure 2, which outlines the Stylus algorithm.

Questions

- Motivation of the selected MLLM - Additional ablation study - Check the quality of adapter descriptions - Details on masking process

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Maybe potential bias and fairness issues will related with this work

Summary

The paper addresses the challenge of selecting and composing relevant adapters for generating high-fidelity images with diffusion models. Stylus introduces a three-stage process: refining adapter descriptions, retrieving relevant adapters based on user prompts, and composing them to create the final image. The paper highlights the development of StylusDocs, a dataset featuring 75K adapters with pre-computed embeddings. Evaluation results show that Stylus outperforms baseline models, achieving higher visual fidelity, textual alignment, and image diversity. The system is efficient and suitable for various image-to-image tasks, including translation and inpainting, demonstrating its versatility and effectiveness in improving image generation.

Strengths

- This is a great paper. Original, high quality, clear, and significant. - The use of adapters and PEFT will/should continue to increase in the future. The authors present a scalable method to improve text-to-image generation.

Weaknesses

None. While adapters have been used in the past to improve image generation, this paper provides a much more coherent strategy to integrate them into VLMs.

Questions

None.

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

None. Great work, Authors!

Summary

The paper proposes Stylus, an approach for automatically selecting and combining fine-tuned adapters on particular tasks to improve the quality of image generation given a prompt. To evaluate Stylus, the paper introduces StylusDocs, a curated dataset containing 75K adapters with pre-computed adapter embeddings. Both the qualitative and quantitative results show that the proposed method outperforms Stable Diffusion and other retrieval methods.

Strengths

- The paper explores an interesting topic of automatically selecting and combining fine-tuned adapters on particular tasks to improve the quality of image generation. - Both the qualitative and quantitative results are proposing. The proposed methods method improves over other method for both human evaluation and automatic benchmarks.

Weaknesses

- Some details about the proposed method are missing, making it hard to reproduce. In particular, sections 3.3 and 3.4 about the composer and the masking are not very clear. How are the adapters selected? How is masking applied?

Questions

No questions beyond the one in the weaknesses.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper does not have any obvious limitations that were not discussed.

Summary

This paper works on post-training optimization for image generation with stable diffusion models. They proposed three stages, refiner, retriever and composer, to personalize a SD model for the prompt and thus to generate the perfect images. The experimental result indicate the potential of the proposed method.

Strengths

1. The motivation makes sense to me and the idea is interesting. Previously, we usually explore prompt engineering to generate a good image, but this paper investigates the adapters and to finalize a suitable model to generate good image given a fixed prompt. 2. The post-training optimization consists of three stages, Refiner, Retriever and Composer, the design of the entire method is reasonable. 3. The experimental results demonstrate the proposed method is promising.

Weaknesses

1. Efficiency. In the paper, the authors compares the Stylus with the typical SD checkpoint, then the efficiency of stylus is not comparable to SD in terms of memory, CPU, GPU resources. 2. Fairness. The paper claim advantages over typical SD in terms of diversity and quality. While it is not quite fair. The Stylus has a model personalization process (from the post-training optimization pipelines), while the SD is using a static model checkpoint.

Questions

1. Does the description (or the optimized version) can well represent the adapter? 2. in Eq.2, why do you directly use betha=0.8? If the 2nd term's value is much bigger than W_base, how do you deal with it?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

All of my concerns have been addressed

Thanks for the clarifications. All of my concerns have been addressed. I am raising the rating from weak accept to accept.

If you have any further questions for the authors, please ask them now

Dear reviewers, Thank you for your continued efforts in helping create the best possible NeurIPS 2024. These reviews are very enthusiastic; I urge you still to check each others reviews and the rebuttals to check whether you have any further questions for the authors -- you have one more day to interact with them. Your AC