Summary
This paper introduces AsyncDiff, an acceleration framework for diffusion models that transforms the traditional sequential denoising process into an asynchronous process. The key insight is that hidden state features in consecutive sampling steps exhibit high similarity. Therefore, feeding the output of the preceding component at time step t-1 as an approximation of the original input to each U-Net component has a negligible impact on performance. This approach allows the parallel denoising processes of AsyncDiff to operate fully asynchronously. Experiments on various versions of Stable Diffusion demonstrate that AsyncDiff significantly speeds up the denoising process for text-to-image generation. Additionally, AsyncDiff is applicable to video generation models such as AnimateDiff and SVD, further showcasing its versatility.
Strengths
1. The writing is overall clear and well-structured. The paper effectively explains the limitations of previous methods that use patch parallelism and introduces a novel approach to "asynchronous" denoising to address these issues.
2. The paper provides thorough comparisons with baseline methods, such as Distrifusion, demonstrating clear improvements in both generation quality and efficiency.
3. The versatility of AsyncDiff is evidenced by experiments conducted on different versions of Stable Diffusion as well as video generation models.
4. The authors address concerns regarding the overhead of communication costs across multiple GPUs, showing that these costs are significantly lower than the model execution time.
Weaknesses
1. Although the main contribution relies on the observation that the hidden states of consecutive steps are similar, the analysis of this phenomenon lacks details. Several key aspects need clearer explanation:
(a) Can the similarity of the hidden states be quantitatively measured? For instance, does a low MSE between hidden states indicate that the two states are “similar”?
(b) Is this phenomenon specific to the U-Net architecture, or is it agnostic to the backbone of the denoising model (e.g., Diffusion Transformer)?
(c) Is it only applicable to DDIM sampling? Does the phenomenon also hold for other fast ODE solvers, such as the DPM solver [1]?
2. While the experiments were conducted using 50 DDIM steps, a comparison with the original model using a number of DDIM steps that achieve a similar speedup would strengthen the argument for AsyncDiff. For instance, in a setup where AsyncDiff achieves a 2.7x speedup, comparing the FID score with the original model using 50 / 2.7 = 19 DDIM steps would clearly demonstrate the necessity of parallelizing the diffusion model.
[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et al., NeurIPS 2022
Questions
Most questions are included in the “Weaknesses” section.
Would a similar asynchronous denoising approach be applicable in a conditional generation setup as well? For instance, achieving faster inference for diverse conditional image generation models, such as ControlNet [1] and Zero-1-to-3 [2], could be practically useful.
[1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang et al., ICCV 2023
[2] Zero-1-to-3: Zero-shot One Image to 3D Object, Liu et al., ICCV 2023
Limitations
The authors discuss the limitations of AsyncDiff (communication cost, dependency on the base model) and the potential societal impacts.