Simple and Fast Distillation of Diffusion Models

Summary

This paper presents a fast-training accelerate sampling algorithm that is based on the distillation paradigm. This algorithm performs trajectory matching under all time points from 1 to 0 (t=80-0.006). The approach avoids the huge overhead of bi-level optimization through the ``detach()'' operation in pytorch, and further improves the performance of the algorithm by correcting the settings of a series of hyperparameters: loss function ($L_1$), step-condition, analytical first step.

Strengths

1. The results of the algorithm presented in this paper are extremely impressive, surpassing all known distillation-based accelerated sampling algorithms. 2. The experiments conducted in this paper are thorough and well-analyzed. Particularly enlightening is the analysis of the guidance scale on Stable Diffusion. 3. The paper is exceptionally well-written, offering clarity and ease of understanding, and it is inspiring in various ways. Notable examples include the introduction of the SFD algorithm and the innovative use of CFG=1 for training, as well as the adaptation of any CFG for inference on SD-v1.5.

Weaknesses

The paper has no significant shortcomings; however, it is worth noting that while the algorithm substantially reduces training costs, it also inevitably lowers the maximum performance threshold. An open question remains: if we extend the training duration, can we achieve results comparable to those of the CTM? It appears that not all time steps contribute equally to model performance. Additionally, introducing randomness through local trajectory matching could potentially enhance generalization.

Questions

1. Can SFD accelerate DiT? It would be beneficial to include some preliminary experiments to explore this possibility. 2. Many accelerated sampling algorithms for diffusion models employ discriminators for trajectory matching, with LCM being a prominent example. Could this discriminator-based approach outperform $L_1$ loss in the context of SFD? 3. Is there a bound for SFD's performance as training time is extended? Given that matching in the form of global inherently involves compression by discarding randomness.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes

Summary

This paper proposes a fast distillation method for diffusion models. This method simplifies the existing knowledge distillation framework and proposes Simple and Fast Distillation from a global perspective to reduce redundant time steps in training. The SFD framework can achieve good experimental results in a very short time compared to existing methods.

Strengths

1. The method proposed in the paper is not complicated and explores many available improvement ideas, achieving good results. 2. Compared with existing methods, the method proposed in this paper can achieve good results in an extremely short time, with significant improvements in time efficiency.

Weaknesses

1. Many of the techniques in the paper lack innovation and are more like technical explorations. 2. Some of the figures in the paper are difficult to understand, and figures like fig.6 should be explained more. 3. The selection of some hyperparameters in the paper, such as t_min, appears too random. 4. If the performance of other training methods such as CTM under the same training time as SFD were reported, it will further demonstrate the superiority of the proposed method.

Questions

1. Add more explanations to some meaningful figs in the paper. 2. Conduct more ablation experiments or explain the criteria for selecting some hyperparameters in the method. 3. Adding more experiment as I mentioned in weakness 4.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations.

Summary

In this work authors propose a novel diffusion distillation method by unrolling the student model to match unrolled/generated pre-trained diffusion (teacher) model's trajectory. Authors demonstrate effectiveness and compute efficiency of approach on stable diffusion.

Strengths

Paper is easy to read and follow. Show effectiveness on stable diffusion and much faster to train for large scale stable Diffusion too. It is interesting to see fixing model at some part improves model over all other parts.

Weaknesses

What is effect of SFD on diversity w.r.t distilled model? It might be easy to have high quality but much lower diversity. Proposed unrolling of student model ( global trajectory optimization) is effectively multi-step training like in structured prediction and imitation learning which has shown to be prone to mode-collapse. Justification of method is still unclear and also not sure on how sensitive is SFD to training time noise schedule/time step weighting i.e., forward diffusion process and its impact on induced trajectory. E.g. if we assume flow matching style linear trajectories, they could be more easier to distill within proposed framework. Unrolling trajectories for distillation is also considered in previous works to account for accumulation of error like BOOT, ImagineFlash, etc. So that in isolation can't be considered as major contribution. Also proposed method though is more effective for Stable Diffusion, there is siginificant performance drop on LSUN, ImageNet etc w.r.t progressive distillation or consistency distillation? This also further raises questions and need for additional experiments to understand under what settings this method is effective.

Questions

Though 'N-model' to finetune has been used, it is interesting to see fixing model at some part improves model over all other parts. This could indicate lower curvature part of trajectory and its more a directional feedback which SFD might be exploiting? Can authors further comment on this? W.r.t trajectory distillation techniques, Consistency Distillation can take potentially shortcuts. But PG retains gradient field. So not sure if such characterization is correct.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

As discussed in weakness and questions generalization to other models is unclear, effect of forward process on SFD? And currently this work also lacks conclusive reasoning under what settings its useful also do not demonstrate impact on diversity, etc.

Most Concerns addressed

I thank the authors for their response. I think most of my proposed concerns are addressed. However, I don't agree with the idea that LCM can't benefit from discriminator. Although the original LCM paper did not leverage GAN Loss, many papers have demonstrated that this form is effective, for example: [1] Kong F, Duan J, Sun L, et al. ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8890-8899. [2] Liu H, Xie Q, Deng Z, et al. SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation[J]. arXiv preprint arXiv:2403.01505, 2024. [3] Zhai Y, Lin K, Yang Z, et al. Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation[J]. arXiv preprint arXiv:2406.06890, 2024. Especially the last one, I've recently explored the effects of the combination of discriminator and LCM, and this pipeline has demonstrated to be very effective. I have scanned the comments of the other reviewers and the authors' responses, and I continue to keep my point that this work is excellent and informative, and I believe that this approach could even be beneficial for dataset distillation on diffusion models. Therefore, I maintain my original score.

Thank you for clarification

I appreciate authors clarifications, would raise my rating to weak accept. It is encouraging to see diversity preserving on CIFAR10, but would be useful to report it on SDv1.5 variants of checkpoints. If precision-recall is expensive to compute/setup, something like LPIPS diversity with say ~30 prompts with 20-30 seeds would be informative for community to understand how this method performs at scale. Agreed on importance of unrolling w.r.t distillation and like the smoothing interpretation of this work. W.r.t Performance drop on LSUN/ImageNet, Question was not why is not SOTA but more about what details w.r.t forward diffusion, weighting etc and its implications on resultant trajectories effect the proposed method. Proposed method robustness w.r.t polynomial coefficients is encouraging, though looking at different class of schedules and resultant trajectories would be interesting, e.g. Flow Matching Schedule vs scaled linear [1]. would encourage authors to consider that for later revision of the paper. Given method has comparable performance SDv1.5 for 4-steps but significant gap to 2-steps compared to progressive distillation albeit at much lesser training. Given method's potential to be more effective if resultant trajectories have less curvature, would consider this work useful and potentially effective for more recent Flow Matching style large models. Would give it weak accept currently, as reasons of when it is effective is a bit unclear to adopt it across settings for broader community. [1] Kingma et al. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

Concerns addressed

Thanks for your response. After reading the authors' rebuttal and other reviewers comments, the concerns are addressed, I raise my score to weak accept. In the next version, I hope this paper could conduct discussion with the concurrent work Relational Diffusion Distillation (RDD). Reference: Feng W, Yang C, An Z, et al. Relational Diffusion Distillation For Efficient Image Generation[C]//ACM Multimedia 2024.