One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance.
Paper
Similar papers
Peer review
Summary
The authors analyze the encoder features of diffusion UNet and find that they share a lot of similarities across certain time steps. Based on this observation, the authors propose to reuse the encoder features and do parallel computation in sampling. This significantly fastens the generation process. Additionally, they also introduce a prior noise injection method to improve the generation quality. Extended experiments demonstrate their effectiveness across a wide range of generation tasks, including T2I, T2V, DreamBooth, controlNet. And their method can be seamlessly combined with various noise schedulers.
Strengths
1 This paper is well-written and easy to understand. The analysis in section 1 is insightful. Fig.4 clearly shows the encoder propagation methods. 2 The empirical evaluation is thorough. The authors validate their method on various datasets, architectures, and use various metrics. They also make a lot of comparison to the SOTA acceleration methods, such as DeepCache. 3 Sec. A4 shows that the parallel denoising leads to an acceptable memory cost, which addresses my concerns. 4 This approach can be seamlessly combined with other acceleration methods such as novel noise schedulers and distillations. Overall, this is an excellent work that significantly reduces diffusion sampling cost in a training-free manner.
Weaknesses
I think the manual selection of key time-steps may not be optimal. Is it possible to use some automatic strategies to determine the key time-steps in your work? Otherwise when we use a new architecture we would have to redo the encoder feature analysis experiment and manually define the key-time steps again. I think there are several works have explored the selection of key time-steps. Discuss on them will make your work even stronger. [1] OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models [2] AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration [3] Denoising Diffusion Step-aware Models
Questions
Please see the weakness section.
Rating
8
Confidence
4
Soundness
4
Presentation
3
Contribution
3
Limitations
Yes, they discuss their limitation in section 5.
Summary
This paper presents an approach to accelerate diffusion model inference by capitalizing on the minimal change in encoder features across time steps. The proposed encoder propagation strategy reduces encoder computations by reusing encoder features from previous time steps. The proposed method is comparable to existing acceleration methods and demonstrates effective acceleration across diverse tasks.
Strengths
1. The proposed method offers a promising solution to speed up diffusion models. 2. The experiments are conducted from various tasks, e.g., text to image; text to video; personalized generation and reference-guided generation.
Weaknesses
1. The evaluation metrics are insufficient, as concerns exist regarding the universality of FID in assessing generative models. It is recommended to validate performance on additional metrics, such as the newly proposed ImageReward [1] and Pick Score [2]. 2. It is unclear whether the latency is comparable when fewer sampling steps are used (Table 11 and Table 12). Moreover, the performance improvement is marginal compared to results with fewer sampling steps, and mainly stems from prior noise injection, which is not the core contribution of the paper. I am also curious whether fewer sampling steps, if equipped with prior noise injection, could achieve similar performance as the results presented in this paper. 3. Comparing multi-GPU results with those single-GPU results from DeepCache seems unfair (Table 1). Besides, the acceleration outcomes on a single GPU are marginally improved when compared with other acceleration methods. 4. The DiT does not inherently include concepts of Encoder and Decoder; thus, categorizing based solely on observations into Encoder and Decoder is somewhat imprecise. 5. The layout of the paper is somewhat disorganized; the order of tables does not align with the sequence of their introduction in the text (e.g., Table 1 and Table 3), making it hard to follow. [1] Xu J, Liu X, Wu Y, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS 2023. [2] Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.
Questions
See weakness for details.
Rating
5
Confidence
5
Soundness
3
Presentation
2
Contribution
2
Limitations
The authors have discussed both limitations and potential negative social impacts.
Summary
This paper presents an extensive study of the evolution of internal activations in diffusion U-Nets and uses their findings to motivate a training-free approach for accelerating sampling from diffusion models. The method is demonstrated to successfully speed up inference in a variety of settings, including different architectures (including non-hierarchical DiTs), different samplers, and some modified inference processes.
Strengths
- The method is demonstrated to work well on a wide range of networks & tasks, including U-Net-based LDMs (SD), Imagen-style cascaded pixel-space U-Net diffusion models (IF), and DiTs. Especially DiT is notable, as previous methods do not work on homogeneous transformer architectures out-of-the-box. - The extensive appendix provides a lot of useful additional information and experimental results and supplements the empirical study about the evolution of internal features in diffusion models well. - The proposed method opens up avenues for further accelerating sampling speed as measured in wall clock delay for single samples by utilizing multiple GPUs to distribute the proposed parallel encoder propagation strategy. - The proposed "prior noise injection method" helps alleviate artifacts incurred from the optimized sampling process.
Weaknesses
- No details are provided about the user study, and false claims relating to it are made in the checklist [questions 14 and 15]. - The main method seems to only be a minor incremental evolution of DeepCache's methodology (encoder features are cached and reused without adaptation of features or network in following steps, optionally multiple times, with the main difference being what is defined as the encoder), and seems to provide a *reduced* speedup when compared in a fair single-GPU setting [Tab. 1, Sec. 4.1 l. 271: 41% speedup vs. 56% speedup, at comparable quality levels]. - The presentation of the paper needs some work. Font sizes in some figures are excessively small [cf. Figs. 1, 2, 3, 7], Tabs. 2 and 3 are excessively compressed up to a point at which readability is impaired. The paper would also benefit from a thorough re-read and corrections for the camera-ready version. Especially grammar errors that substantially affect the claims made in the paper (e.g., sampling time reduction "to 41%" [l. 271] instead of "by 41%") and typos that change the meaning of words (e.g., "rudely" [l.105]), but also names of methods (e.g., "DeepFolyd-IF" [l. 60], "Stable Diffuison" [l. 525]).
Questions
- Can the authors provide examples and evidence of practically relevant settings where DeepCache fails or underperforms significantly compared to the proposed method (such as a combination with DPM-Solver(++), ControlNet, or Dreambooth etc)? These would help address the main method-related weakness mentioned. - What do the ellipses in the key time step steps mean? Is every time step between 80 and 25 included for $\{80,...,25\}$? - Is there a relevant difference between the proposed "prior noise injection method" and the various methods that artificially increase the noise level slightly during ODE sampling, such as ``churn'' in (Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022)? - Are all the videofusion samples in the supplementary material just corrupted or is there something to be demonstrated there? - Does parallel-batch encoder propagation limit the possible batch size on a single GPU compared to other sampling methods? If so, could the authors provide a performance comparison that individually maxes out the batch size for the various approaches (including competing ones) and then reports the samples/second? As generating multiple variants simultaneously is a standard approach when generating images with diffusion models in practice, this is an important avenue for optimization. Specifically, I am worried that at a maximal batch size, parallel encoder propagation will only offer a negligible speedup compared to encoder propagation (due to larger generation batch sizes being possible with non-parallel encoder propagation, as VRAM usage is substantially different between the two methods [Tab. 8]), at which point this method would provide less than half of the speedup provided by prior art.
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
2
Limitations
The authors have adequately addressed the limitations of their work.
Summary
This method can be applied at inference time without requiring retraining to improve sampling efficiency while maintaining high image quality and can be combined with other approaches to speed up diffusion models. The approach is effective across various conditional diffusion tasks, including text-to-video generation (e.g., Text2Video-zero, VideoFusion), personalized image generation (e.g., Dreambooth), and reference-guided image generation (e.g., ControlNet). Using encoder features from previous time steps as input for decoding multiple later time steps allows for concurrent decoding, further accelerating SD sampling by 41%, DeepFloyd-IF sampling by 24%, and DiT sampling by 34%. Besides, a prior noise injection strategy is introduced to preserve image quality, maintaining texture details in the generated images.
Strengths
Novelty: The three techniques to design efficient diffusion models are novel. Significance: The problem of efficient diffusion transformer inference acceleration is significant in diffusion models. Methodology: The proposed algorithm is well-formulated and well-explained. Results: The experimental results show significant improvements over existing methods over SD, DiT, and DeepFloyd-IF and are widely applied to text-to-image, text-to-video generation, and personalization.
Weaknesses
The visualization results for smaller steps appear suboptimal, potentially due to the absence of consistency models or adversarial distillation models for verification. This raises concerns about the foundational assumption of maintaining high similarity throughout the process. It is crucial to explore whether incorporating consistency models or adversarial distillation methods could enhance the quality of results and ensure the assumption of high similarity holds even in smaller steps. Providing more robust verification methods would strengthen the validity and reliability of the visual outputs generated by the proposed approach.
Questions
Extra Cost to Identify Key Time-Steps: What are the additional costs associated with identifying key time steps in the diffusion process? Is this identification process capable of being learned dynamically, or is it independent of the data? Understanding the computational overhead or resource allocation needed for this identification process is crucial for evaluating the overall practicality of the method. Enhancing Application Area with Smaller Steps Diffusion Models: Applying this approach to smaller steps diffusion models has the potential to significantly expand its applicability. By effectively handling smaller time steps, the method could be utilized in a wider range of scenarios, enhancing its versatility and impact.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Please refer to the weakness and questions part.
> **W1. User study details.** > > The study participants were volunteers from our college. The questionnaire consisted of 35 questions, each presenting two images: one from the baseline methods (including ControlNet, SD, DeepFloyd-IF, DPM-Solver++, Dreambooth, Custom Diffusion, Text-to-Video, etc.) and the other from our method FasterDiffusion (one example shown in Figure.29). > > Users were required to select the image where the target was more accurately portrayed, or choose “both equally good”. A total of 18 users participated the questionnaire, resulting in totally 630 samples (35 questions × 1 option × 18 users). As the final results shown in Figure.8, the chosen percentages for SD, DeepFloyd-IF, DPM-Solver++, Text-to-Video, ControlNet, and Personalize were 48%, 44%, 51%, 52%, 46%, and 39%, respectively. These results show that our method performs on par with the baseline methods. Thanks for providing these details and the screenshots in the rebuttal PDF, but this does still not resolve the major concerns about question 15 (IRB board approval, potential risks for participants).
IRB considerations
I appreciate your emphasis on the significance of IRB approvals for user studies in current computer vision research. It is indeed crucial to ensure that research involving human subjects adheres to ethical standards and guidelines, and obtaining IRB approval is a key step in this process. The Institutional Review Board (IRB) is a formalized entity tasked with upholding research ethics by conducting a rigorous assessment of the methodologies proposed for research endeavors that involve human subjects. The primary objective of these reviews is to safeguard the wellbeing of study participants, ensuring that they are not subjected to harm and that their rights and welfare are duly protected. This is achieved through a meticulous examination of research protocols and associated documentation, with the intention of identifying and mitigating potential risks. In the context of this research paper, it is noteworthy that the study design does not pose any risks of harm to the participants. Instead, we employ a user study methodology wherein participants are invited to express their opinions regarding the images generated by our approach in comparison to those produced by alternative methods. To further safeguard the participants, we have implemented rigorous safety checks within our image generation pipelines, ensuring that no harmful or inappropriate content is included in the output images. Consequently, the study adheres to ethical principles by minimizing potential risks and prioritizing the protection of participant rights and welfare. In this context, we address checklist item 15 as outlined by the regulatory body, as our study does not involve research concerning human subjects. Rather, it may be construed as an online questionnaire administered for data collection purposes. We endeavored to address all your concerns comprehensively to ensure clarity in the explanations provided. If you have further questions about our research, please feel free to engage with us for further discussion.
Given the authors' responses to my concerns about the user study, my main concerns relating to it and the missing material in the paper have been resolved. To the best of my knowledge, the user study performed by the authors qualifies as "research involving human subjects" in most jurisdictions, but this might not be the case in theirs, and I am no ethics reviewer who would have the qualification to judge in a more precise manner. I would still urge the authors to double-check that they followed all the rules set by their institutions, country, and NeurIPS, and expect them to include all relevant details, especially those presented as a part of the rebuttal, in the final version.
After carefully reviewing the authors' rebuttal and review responses, only some of my concerns have been resolved. One important remaining concern is comparisons in fair settings. The authors were able to address my concern about a reduced single-GPU speed compared to the previous DeepCache in the case of using ControlNets, but not in general. More importantly, as far as I could see, the authors still did not provide any results that max out the batch size per GPU, as suggested in my initial review. I think this is an essential comparison, as VRAM usage is one of the main practical limitations for practical deployments, both due to the cost of GPUs and, more importantly, because larger batch sizes typically make inference more efficient. This is relevant even if only single images are generated per user, as any standard inference backend will batch multiple calls for efficiency. The fact that the method proposed in this work shows a substantial VRAM increase would suggest that this method's speed is likely limited if one increases the batch size to the maximum possible on a reasonably sized GPU. Additionally, I disagree that the authors' argument regarding multi-GPU inference is practically relevant. Many practical inference systems generate multiple images at once (at which point one can just split up the samples over multiple GPUs instead of splitting up the inference process of each sample over multiple GPUs). In other cases, [unless one is already using a top-of-the-line available GPU, which is heavily non-standard for diffusion production loads], one can likely increase the inference speed by a similar amount at the same cost by just using one higher-end GPU instead of two GPUs. I was now also able to open the problematic Videofusion example videos from the supplementary material on a Windows machine using the standard media player. My issues with opening them persist on the two different MacOS-based systems I tried opening them with initially (using the standard viewer as suggested by the authors). If the authors are aware of any uncommon codec choices etc that might have affected this, I'd recommend that they switch to a standard encoding for future submissions to prevent these problems. Most importantly, I have been able to look at the samples now, and they seem reasonable. As the authors have addressed my major critical concerns about the user study and some minor concerns but left my major concerns about fair comparisons insufficiently addressed, I'm raising my score from 3 to 5. With aforementioned shortcomings, I think the main value this work provides in its current form is the analysis of the network, but I'm sceptical about the practical relevance of the proposed method to exploit them in general use cases.
We are grateful for your prompt response, and we are glad to clarify and address your concerns further. **User study details and its IRB concerns** Thanks for your attention to the user study, particularly in relation to the IRB rules. In any future version, we are committed to incorporating detailed experimental information on the user study accompanied by a thorough explanation of IRB approvals, which we already addressed in the rebuttal phase. **Single-GPU inference time comparison** To further relieve your concerns over the single-GPU setting, we conducted experiments under the same setup. Our observation indicates that SD1.5 model attains a peak batch size of 30, with an average inference time of 1.903 seconds per image. Upon integration with our approach FasterDiffusion, the maximum batch size experiences a minor reduction to 25, but the inference time per image demonstrates a significant decrease to 1.551 seconds. These results highlight that, despite the constraints of the maximum batch size per GPU, our method delivers a notable enhancement in inference speed. FasterDiffusion presents a robust solution for navigating time-space trade-offs effectively. **Video format issue** We appreciate your valuable feedback and the efforts you made in examining our video examples. We will investigate this matter thoroughly and guarantee the application of standard encoding in any future submissions. We are glad to hear that you were able to review the samples and found them reasonable. In the future version, we aim to release a project page where all examples, including the Videofusion samples, are easily accessed for convenience.
Thank you for providing the additional evaluation for the max batch size setting. In my opinion, the presented speedup of ~20% (down from the 41% presented in the paper, which was already lower than that of other methods) in this fair evaluation setting is not particularly favorable for the method proposed in this work. Thus, I will retain my initial increased score.
Thank you for your response. I'm still a bit confused, as typically, the PickScore should be a specific numerical value, not a percentage.
Dear Reviewer wo3A: We are grateful for your expeditious response and we are glad to clarify and address your concerns further. For the PickScore, we used the calculation method recommended by the official github repository reference: > For a prompt $x$ and a image $y$，the scoring function $s$ computes a real number by representing $x$ using a transformer text encoder and $y$ using a transformer image encoder as d-dimensional vectors, and returning their inner product: $$ s(x,y) = E_{txt}(x) · E_{img}(y) · T $$ Where $T$ is the learned scalar temperature parameter of CLIP. Following that, assuming the original image generated by SD is $y_1$, and the image generated after combining with Ours is $y_2$, we can calculate the preference distribution vector $p$: $$ p_i = \frac{\exp s(x, y_i)}{\sum_{j=1}^2 \exp s(x,y_j)} $$ PickScore official recommended get probability, if there are multiple images available for selection. We also calculated the original PickScore scores, as shown in the table below: | | PickScore$\uparrow$ | s/image | | ---------------- | ------------------- | ------- | | SD(DDIM) | 21.43 | 2.42 | | SD(DDIM) w/ Ours | 21.35 | 1.42 | Our method FasterDiffusion enhances sampling efficiency while preserving the original model performance. We hope that these clarifications have addressed your concerns satisfactorily and look forward to any further feedback you may have. We are committed to enhancing our work based on your expert guidance.
Thank you to the authors for the detailed response. Most of my concerns have been addressed, so I have raised my score.
Decision
Accept (poster)