Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis

Summary

This paper introduces Hyper-SD, a novel framework designed to mitigate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). The proposed framework addresses the limitations of existing distillation techniques, which focus on either ODE Trajectory Preservation or ODE Trajectory Reformulation and often suffer from performance degradation or domain shifts. Hyper-SD synergistically combines the advantages of both approaches while maintaining near-lossless performance during step compression. Key innovations include Trajectory Segmented Consistency Distillation for preserving the original ODE trajectory, human feedback learning to enhance low-step performance, and score distillation to improve low-step generation capabilities. The framework also uniquely leverages a unified LoRA for all inference steps. Extensive experiments and user studies demonstrate the Hyper-SD's supreme performance 1 to 8 inference steps for both SDXL and SD1.5, surpassing existing methods such as SDXL-Lightning in CLIP and Aes scores.

Strengths

1. The paper comprehensively considers segmented distillation and the incorporation of human feedback scores, potentially outperforming the original diffusion models in certain cases. It even enables one-step image generation. 2. The experiments are thorough and well-executed and the paper is well-writen.

Weaknesses

1. The additional introduction of the human feedback model might result in excessive memory usage. Since the evaluation steps require decoding and an additional model, would the actual computational resource consumption be clarified? 2. The inclusion of reward terms in the loss functions of LCM-based methods has been explored in related works, such as Reward Guided Latent Consistency Distillation. A comparison with these methods, if feasible, would be beneficial.

Questions

refer to weakness

Rating

3

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

refer to weakness

Summary

The paper introduces a novel framework named Hyper-SD, which enhances diffusion models’ efficiency by combining consistency distillation, human feedback learning, and distribution matching. Hyper-SD performs consistency distillation in segments to preserve the original ODE trajectory, maintaining high-quality generation while reducing inference steps. Human feedback learning further boosts performance in low-step inference, mitigating performance loss during the distillation process. Experimental results demonstrate that Hyper-SD achieves state-of-the-art performance from 1 to 8 inference steps, significantly surpassing existing methods, particularly in aesthetic scores and CLIP scores.

Strengths

1. The components of this work are clear: a) Based on TCD, the author segments the ODE and then performs distillation on these segments. b) To enhance the model’s overall performance, the author utilizes RLHF techniques. c) The author further employs the DMD scheme to optimize the results obtained in a single step. 2. The results are solid, clearly demonstrating the effectiveness of the proposed approach.

Weaknesses

The primary issue is the insufficiency of the ablation experiments, the whole Hyper-SD sounds contain 2 modules, a TCSD LoRA, a RLHF LoRA, and TCSD LoRA for 1 step needs further DMD loss. However, the ablation study does not demonstrate the effectiveness of each part. Here's my concerns: 1. In TCSD part, the distance metric includes GAN and L2. However, in TCD, it only contains L2. Is this metric similar to CTM (L2+GAN) or different? And how's the contribution of GAN part? 2. RLHF module needs another LoRA, is this the only different between Tab 3 w/ and w/ RLHF? 3. May I know the RECALL of the module w/ (w/o) GAN, w/ (w/o) RLHF ? 4. RLHF module is totally separately trained from the TCSD module? 5. The one-step model is the same training pipeline with TCSD (which means [8,4,2,1] + RLHF) but with DMD loss based on some base models like SDXL? Or it is a continue trained model based on a TCSD LoRA? 6. For one-step generator for comparison, does it include RLHF LoRA? 7. A minor question. Is $\Delta t$ ( or we can say $t_n - t_{n-1}$ ) the same for different segs? My feeling is it should be same, but in Fig.2, the length of ODE solver is obviously different.

Questions

Please check Weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

the limitations are discussed

Summary

This paper presents an approach for distilling a diffusion model into a multi-step generator. Previous distillation methods typically fall into two categories: those that preserve the ODE (Ordinary Differential Equation) trajectory and those that match the teacher model at the distribution level. This research offers a unified solution by integrating ideas from both categories. Specifically, it employs a multi-segment consistency distillation objective, akin to the ODE trajectory-preserving methods like Consistency Trajectory Model, and incorporates a combination of GAN (Generative Adversarial Network) and distribution matching distillation losses from the distribution-level matching methods. Additionally, an important contribution of this work is the incorporation of human or reward model feedback in the distillation process. The resulting method demonstrates high-quality text-to-image generation.

Strengths

S1. The paper is well-written and easy to follow, providing a thorough introduction to the background and connections to previous works. S2. The final approach is effective, with solid ablation studies of various components. The evaluations are comprehensive and demonstrate strong results. S3. The utilization of reward optimization in diffusion distillation is relatively new, showcasing the potential of unifying distillation with other post-training techniques to enhance final performance in terms of aesthetics, alignment, and efficiency.

Weaknesses

W1. The final method combines well-explored modules (consistency trajectory model, GAN, and DMD) into a unified framework. While this effective integration is a contribution, it is not particularly novel and doesn't introduce significantly new knowledge. W2.Regarding the evaluation, diversity comparisons are missing from the current paper. Although FID is not the best metric, the authors should still report it for a comprehensive understanding. Additionally, assessing per-prompt diversity is important. The authors are encouraged to generate a set of images with the same prompt and measure their variations as a proxy metric. I am particularly interested in the influence of reward optimization on generation diversity. W3. For one-step generation, the improvements appear small compared to vanilla DMD (Table 3).

Questions

My concerns are outlined in the weaknesses section. While the paper demonstrates strong results and well-executed experiments, the overall method lacks significant novelty or performance improvement. Therefore, I will give it a borderline accept as my initial rating.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper studies the distillation problem of diffusion models. Specifically, it introduces Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory. Besides, the human feedback learning and Distribution Matching Distillation (DMD) technique are also included in the proposed model.

Strengths

+ This paper is well written and easy to follow. + The studied problem is interesting and the proposed technique follows up the previous works to extend to a more general training strategy. + The user study is informative to demonstrate the superior performance of the proposed model.

Weaknesses

- The proposed method of TSCD sounds interesting but some details may be questionable. First, intuitively, the randomness of time segments and the choices of t_end may introduce the randomness in the model training process. How can we guarantee the model will always converge to the same results given these different design choices? Second, comparing with CTM, how can we guarantee the multi-step training with different time steps can essentially introduce the advantages? For example, the multi-stage training may also introduce accumulated errors. - It is interesting to add human feedback as a guidance. The paper also mentioned that human feedback learning may distort the output distribution. This is quite important. How can we know how human feedback learning is changing the distribution? How to detect and evaluate that? It is not very obvious why employing the LoRA merge technique with the TSCD can achieve a flexible balance between generation quality and output domain similarity. Can the author elaborate more on this? Any experiments or evidence to support this claim? - Concern about the technical novelty. The proposed model consists of multiple techniques that have been introduced and commonly used in previous works. Although the performance looks promising by adding up all these modules together from the user study and the proposed method looks reasonable, there may not be some essential out-of-box ideas or findings that are very exciting. - More detailed ablation study would be helpful. It is appreciated that the author conducts the ablation study to evaluate the effects of different modules. However, it is desirable to have a more thorough ablation study besides Table 3, since it may be still not very clear which modules among the three components play the most important factor in the proposed method. For example, without human feedback, the performance of TCD and TSCD is quite close, especially with more steps. How about with human feedback to TCD and then compare it? Does DMD are added in both TCD and TSCD experiments to have a fair comparison?

Questions

Please see the weaknesses for specific questions.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The author discusses the limitations of the proposed method in the appendix, although it focuses more on the potential future works and follow up directions, instead of limitations and potential negative societal impact of their work.