Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

Summary

The paper introduces a novel method called SPIN-Diffusion for fine-tuning text-to-image diffusion models. SPIN-Diffusion uses a self-play mechanism where the model competes against its earlier versions to iteratively improve its performance. This approach eliminates the need for human preference data, which is a significant requirement for traditional reinforcement learning-based fine-tuning methods. The experiments on the various dataset demonstrate that SPIN-Diffusion outperforms existing supervised fine-tuning methods and reinforcement learning-based approaches, achieving higher human preference alignment and visual appeal with less data.

Strengths

S1: The paper presents an innovative self-play fine-tuning method that does not rely on human preference data, addressing a significant limitation in current fine-tuning approaches. S2: Extensive experiments are conducted, showing that SPIN-Diffusion outperforms both supervised fine-tuning and reinforcement learning-based methods in terms of human preference alignment and visual appeal. S3: The theoretical analysis provides a strong foundation for the proposed method, demonstrating its convergence and superiority over traditional supervised fine-tuning methods. S4: The paper effectively communicates the technical challenges and solutions, making the methodology accessible to readers.

Weaknesses

W1: The paper lacks a comparison with traditional fine-tuning methods for diffusion models, e.g., LoRA. W2: The computational overhead of the self-play mechanism is high, requiring 5-10 times more training time compared to baselines, which might limit its practical application. W3: The motivation of this method is unclear to me. It would paint a more holistic picture of the problem and solution. W4: The paper assumes that the data distribution can be adequately represented by the parameterized family, which may not hold in all practical scenarios. W5: The evaluation is primarily focused on a single dataset (Pick-a-Pic), and additional benchmarks could strengthen the generalizability of the results. W6: The font size of Figure 1 is too small.

Questions

- Why should fine-tuning the DMs use RL methods? Could you please discuss the advantages of fine-tuning DMs with RL compared to traditional methods? - As the authors claimed, "In many datasets including the community-sourced ones featuring custom content, it is often the case to have only one image associated with each prompt. This makes RL fine-tuning infeasible."I do not understand why this approach is considered infeasible. In my prespective, one image and its prompt can be considered as the observation of agents. The other processes are similar to the standard RL paradigm. Therefore, it would be greater to discuss more.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to weaknesses.

Summary

This paper introduces a method called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the model engages in a competitive process with its earlier versions, driving iterative self-improvement. This method presents an alternative to conventional supervised fine-tuning and RL strategies. Experimental results on the Pick-a-Pic dataset show that SPIN-Diffusion outperforms previous supervised fine-tuning in human preference alignment and visual appeal. Key contributions of this work include the introduction of SPIN-Diffusion and its empirical validation demonstrating superior performance compared to existing fine-tuning methods.

Strengths

1. The paper for the first time applies SPIN to diffusion model to my knowledge. 2. Theoretical analysis shows that the proposed approximate SPIN loss is an upper bound of exact SPIN loss.

Weaknesses

1. The main contribution of this work is an approximate SPIN loss compared to the previously proposed exact SPIN loss. The major modification is moving the average over sampling steps $t$ outside the loss function, resulting in an upper bound. I kindly argue this improvement is straightforward when transferring SPIN from LLM to diffusion model, without much insights into diffusion model itself. 2. Since the approximate SPIN loss is proposed for data/memory/time efficiency, I didn't see any computing efficiency comparisons of the approximate and exact SPIN loss in the main paper or supplements. As noted by the authors, "it requires additional sampling overhead, approximately 10 times of the training time when using traditional DDIM sampling." This additional computing cost significantly limits the method in practice. 3. Assumption 4.1 assumes the loss $l(t)$ being monotonically decreasing and convex. It seems like the SPIN loss proposed in Equations. 3.8 and 3.9 do not meet this strong assumption. It makes the analysis results less convincing. 4. Quantitative performance improvement (Table 1) is mild. 5. It's hard to tell which method generates the best pictures in qualitative comparison (Figure 2).

Questions

1. The "(ours)" notations in the tables are a bit confusing, may misleading the audience considering them as a method proposed in this work. --- After rebuttal: Some of my initial concerns have been addressed, while I still hold concerns on the method insight, time efficiency, and quantitative improvements. By comprehensively considering the cons and pros of this paper, my final raiting would be borderline accept.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

- This paper introduces SPIN-Diffusion, a new self-play fine-tuning technique for diffusion models that improves iteratively by competing with previous versions. - They show that SPIN-Diffusion outperforms existing supervised and reinforcement learning fine-tuning methods in aligning with human preferences and enhancing visual appeal starting from the first iteration. - This method is more data-efficient, achieving superior results with fewer data, which is beneficial in cases where there are limited images available per text prompt. - The paper uses a competitive setup between two iterations of the model to generate and evaluate images, considering all generated images in the evaluation process, not just the final product. - Their experiments on the "Pick-a-Pic" dataset demonstrate that SPIN-Diffusion consistently surpasses other methods in multiple performance metrics through successive iterations. - The approach is cost-effective and offers a practical solution for improving diffusion models, particularly useful in environments with restricted data access.

Strengths

- The extension of SPIN to diffusion models is well-formulated for the problem at hand. - Theoretical explanations are detailed and well-supported. - The design of an approximate version of the objective function, considering computational efficiency, appears practical. - The method shows practical utility by outperforming previous methods that required "loser" samples within a few iterations.

Weaknesses

- The sampling overhead is significant, requiring 5-10 times more training time. - There is insufficient explanation regarding the assignment of the hyperparameter \( \beta_t \) and its variation across iterations. - If a stronger approximation is applied and the sampling overhead is reduced by focusing on trajectories rather than all time steps, it might be possible to compare improvements with significantly reduced training time. Showing improvements within a setup that grants no more than twice the training time compared to SFT could have demonstrated the efficacy of SPIN-Diffusion more clearly.

Questions

- I would like to see a practical comparison of computation time in GPU hours between SPIN-Diffusion and other methods.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

- The sampling overhead is significant, requiring 5-10 times more training time.