Parallel Sampling of Diffusion Models

Summary

The paper introduces a new method for speeding up (reduced latency) sampling of diffusion models by sampling all time steps in parallel from some initialization and then iterating this procedure until convergence (Picard Iteration). Modifications are made for implementation efficiency such as a sliding window. The method is evaluated on control tasks and large scale image datasets with a consistent reduction in latency at the expense of increased compute being reported.

Strengths

This paper is very good. The problem being tackled is highly significant and I think the approach will be widely used to easily achieve a reduction in latency when extra compute is available. The method is clearly presented with a nice bit of theoretical analysis. The experiments on practically releveant datasets clearly show the benefits of the method. I highly encourage acceptance.

Weaknesses

I think it would have been nice to pick a suitably small problem and fully explore the compute/latency tradeoff by varying the window size. I appreciate the current experiment in Figure 4 which does nicely show this trade-off in this compute intensive regime however the picture is slightly obscured by the necessity of also considering GPU flop/s saturation (which is an important consideration in its own right I agree). I think a simpler experiment where you can vary the window size all the way up to the number of denoising steps would benefit the paper where we can see a potential 20x reduction in latency if DDPM is integrated with 1000 steps and the Picard iterations converge in around 50 steps. For the experiments on StableDiffusion, it is not clear to what extent the method's benefits extend into the very low model evaluation regime. DPMSolver should be able to go to very few model evaluations and retain decent quality, in this regime, how does your method compare? It is ok if the model speedup is less in this case, but it would be good to know the extent to which the method can provide benefits. I think it would be good to stick with the terminology batch window size as you go through the experiments. When getting to section 4.2.1 it is a little confusing to start talking about batch sizes which may imply that you are generating entirely independent trajectories in parallel but I assume you still mean the batch containing the points in the window. Edit after rebuttal: I have read the author's response and my concerns have been addressed. I will keep my score of 8.

Questions

In proposition 2, how realistic is the assumption of the convergence rate of the Picard iterations in the context of integrating the probability flow ODE from a trained diffusion model? This has been investigated somewhat between the two experiments but what would you say is the interaction between the number of Picard iterations and the number of steps used to integrate the ODE. Do you believe there could be any relation where more steps could be helpful to reduce the number of total Picard iterations or are these completely orthogonal quantities?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The method is obviously more expensive than other diffusion models due to all steps being taken in parallel but this trade-off is well explained. Further, as explained above, the limits of when the method does not actually provide a speedup could have been better investigated.

Summary

Instead of reducing the number of denoising steps, the paper proposes to parallelize diffsuion denoising sampling via Picard iterations, by guessing the solution of future denoising steps and iteratively refining until convergence, which trades compute for speed. The authors then present ParaDiGMS to accelerate the sampling of pretrained diffusion models by denoising multiple steps in parallel and verify its effectiveness.

Strengths

- The idea is interesting and may promote another way for accelerated sampling. - The procedure of Algorithm is clear.

Weaknesses

In total, I think there are still many problems for the paper. Please refer to the questions.

Questions

**Q1**: Some notations may be a bit confusing, since the $x_T$ is always used to represent the initial noise of diffusion and the authors use $x_0$ to represent the initial point. **Q2**: For Algorithm 1, given the same denoising steps $T$, the proposed method is essentially similar to the existing techniques such as PNDM, DPMSolver. The existing high-order techniques should also cache the predicted score in previous time step, while the proposed method compute all the predicted score in a small window and use them for next selected window. **Q3**: The number of model evaluations: For all the tables, do you choose a different $T$ and use more parallel iterations compared to the baseline to match the sampling performance? If so, what is the value of $T$ ? If I am not wrong, $T = Model Eval / Para Iter$ ? The details are not clear. **Q4**: Following the prior question, how about setting a same $T$? Can parallel sampling improve the performance (e.g. FID)? This results could be important. If the performance remain the same, people can directly use a smaller denoising step $T$. **Q5**: For image generation, does $batch size = 80$ mean the author uses a batch window size of $80$? **Q6**: The evaluation results of image generation are not sufficient. How about the commonly-used FID for stable-diffusion? And more qualitative results should be shown and analysed. **Q7**: The algorithm seems not practical. As the authors said, this method may consume more resources. However, **Q4** is not clear. If the performance remain the same, more resources should be used to maximize sample throughput instead of parallel computing. **Q8**: Related works should discuss more parallelization techniques and the differences of between the proposed method and similar parallelization techniques.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

The paper proposes an approach for speeding up sampling of diffusion models using Picard iterations. Multiple time steps in the sample process are predicted in parallel, iteratively refining until converging. Rather than refining all time steps at once which would not be practical, a sliding window approach is used: points in the window are refined then the window is moved once the later time steps have converged. Experimentally this approach is tested on diffusion policy learning and image generation benchmarks, where it is shown to provide substantial speedup over sequential sampling methods, while achieving comparable sample quality.

Strengths

- Using Picard iterations to speed up diffusion model sampling makes sense and is useful in many low latency scenarios such as image editing. - Showing that in the worst case, the approach will converge faster at least as fast as sequential sampling is nice (Lines 127-137). Similarly, providing tolerance bounds is useful (Lines 161-164). - A good set of experiments/ablations on a variety of benchmarks are provided. The approach is clearly shown to be faster than sequential approaches, while providing similar quality samples. - The approach is tested with multiple diffusion samplers, including the recent DPMSolver, showing that it can provide speedup even to already fast solvers.

Weaknesses

- It is mentioned that a more relaxed tolerance value can be used when determining convergence (line 166). It would be useful to see the effect of that value on sampling times/image quality. - Currently multiple GPUs are required to achieve net speedup on Stable Diffusion (Figure 4f). - The approach is only useful for reducing latency; if generating lots of samples, then sampling batches sequentially is more efficient. However, as mentioned in the strengths, there are many applications where this is useful.

Questions

- In Algorithm 1 on line 9, new points are initialised from the latest point in the window. Would it make more sense to initialise the values based on the prediction of $x_0$? - What practically is the impact of using different tolerance values?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are well discussed in section 4.2.1 and section 5. It’s also worth noting that due to the extra compute, this approach will use more energy so has a negative environmental impact.

Summary

The authors propose a technique to reduce the time taken to sample from a diffusion model at the expense of using more FLOPs. Roughly speaking, the authors parallelize sampling by "guessing" xt at numerous values of t simultaneously, then computing the score function for each of these values of xt in parallel, and then repeating to refine the estimate of each xt over multiple steps.

Strengths

- The paper is clearly written. - The proposed method is novel. - The proposed method is practically useful in situations where having low latency is important (e.g. iterative human-in-the-loop image generation). - The experiments are thorough, showing speed-ups on a range of domains and even showing speedups when efficient samplers like DPMSolver are used.

Weaknesses

Overall I am satisfied that this paper makes a tangible contribution. I have a couple of questions though about certain scenarios in which it is not immediately clear that this method is useful. - I would be interested to see the performance of the latent image generation when less steps are used. E.g. for Stable diffusion (https://github.com/CompVis/stable-diffusion), the given example command "python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms " produces good images in only 50 steps, which is much less than the 200 used in Section 4.2.1. Does the proposed method still provide a gain in this setting? - How does this method interact with other techniques to speed up sampling like progressive distillation (https://arxiv.org/abs/2202.00512)? I appreciate that one benefit of the proposed method is that it avoids the need to have any sort of "distillation" training phase, but can the proposed method provide additional advantages when used in combination with a "progressively distilled" model?

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are adequately addressed.

Summary

Naive parallelization can let us generate multiple samples which improves throughput. However, the wall-clock time remains the same. To reduce the wall-clock time during diffusion model sampling, this paper proposes ParaDiGMS, a parallel sampling method of diffusion models based on Picard Iteration which improves latency. ParaDiGMS is compatible with classifier-free guidance and prior fast sampling methods. The experiment results show that ParaDiGMS can achieve a speedup without quality degradation.

Strengths

1. The authors provide a theoretical guarantee and error analysis of the proposed method. 2. The paper is motivated by the classical Picard Iteration which brings new ideas to the diffusion sampling community. 3. The paper is easy to follow and overall well-written which helps the reviewer understand the content. 4. The experiment results show that it achieves a speedup with no measurable decrease in quality.

Weaknesses

The applicability of the proposed method is limited to certain situations. The total evaluations of ParaDiGMS are approximately twice that of the baseline method according to the experiments. Therefore, the proposed method may not be suitable for situations where maximizing sample throughput is a priority.

Questions

In the section on experiments of diffusion image generation, the authors test ParaDiGMS on the settings of 1000 NFEs for DDPM, 200/500 NFEs for DDIM, and 200 NFEs for DPMSolver. It is questionable whether ParaDiGMS can achieve a similar speedup in fewer NFEs(like around 50 NFEs) since fewer NFEs are commonly used in downstream applications and the state-of-the-art samplers can achieve comparable results in this setting.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author addressed the limitations of their work.

Response to Authors

Thanks to the authors for their responses; it is great to see the improvement in the low step scenario, this definitely strengthens the approach making it applicable in more scenarios. Additionally, I appreciate the evaluation of the tolerance value, I found this very informative. And I thank the authors for the comments on different initialisation strategies, it is interesting to hear that it is not always helpful. After these comments and the responses to the other reviewers, I am happy to increase my rating to accept.

Thanks for the responses

Thanks for the responses. Most of questions have been addressed. And I believe that the idea is interesting and can bring new direction to the diffusion sampling community. However, - Besides CLIP score, FID is also a common metric for text-to-image to evaluate the image quality. The evaluation is usually done in a zero-shot setting on text-image datasets of natural images such as COCO. The readers may concern about the image quality as well. - When it coming to the balance between latency & performance, one would expect higher performance with lower latency. As we known, reducing the number of sampling steps could be an obvious way to reduce the latency. Therefore, the comparison(e.g. FID、CLIP score) of different methods with the same latency is important, too.