DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics

Summary

The paper proposes a new sampling algorithm, DPM-Solver-v3, to solve the diffusion ODE. Applying the Rosenbrock-type exponential integrators, DPM-Solver-v3 pre-computes several coefficients (empirical model statistics) to minimize the norm of the gradient of non-linear term, which leads to reduced discretization error. Empirical results of DPM-Solver-v3 demonstrate its effectiveness in unconditional/conditional sampling and classifier-free guidance on Stable Diffusion.

Strengths

- By introducing empirical model statistics arising from Rosenbrock-type exponential integrators, DPM-Solver-v3 has better model parametrization than data/noise prediction ones, which leads to lower discretization error. - Experiments show that DPM-Solver-v3 outperforms DPM-Solver++ and UniPC for both conditional and unconditional sampling. DPM-Solver-v3 also incurs a smaller MSE on Stable Diffusion, across NFEs.

Weaknesses

- The sampling algorithm is based on the pre-computed empirical model statistics $l_{\lambda}, s_{\lambda}, b_{\lambda}$ at each time step $\lambda$, and their corresponding estimated integrals. This seems to lead to extra memory cost and makes the model not able to be flexibly adapted to new use cases (e.g., a different time schedule, guided sampling, etc.) - When developing the error order and convergence of the algorithm in section 3.2, the estimation error of EMS is not taken into account. However, the unstable $s_{\lambda}$ w.r.t $\lambda$ in Figure 5 (especially for the one estimated on ScoreSDE) indicates that the EMS estimation error is not ignorable. Is it possible to provide such analysis or guarantees given the error of EMS? - In the low FID region (which is closer to practical usage), the benefit of DPM-Solver-v3 seems not very significant compared with UniPC in Figure 3 and Figure 4 - Applying Rosenbrock-type exponential integrators to diffusion ODE is a valid contribution, but other components (e.g. higher-order derivative estimation) of the method seem to be quite relevant to those proposed by existing works like the RK45 Butcher tableau or [1,2]. The real improvement of the Rosenbrock-type technique is unclear when compounded with many other components. Could the authors maybe ablate on its individual effect? - I think it would be very helpful to compare the FID score / CLIP score on Stable-Diffusion since low MSE does not necessarily imply better sample quality (the quality is affected not only by discretization error but also by estimation error of neural networks). - Could the authors also compares DPM-Solver-V3 with some typical stochastic samplers, like Alg 2 in EDM, Gonna Go Fast [3], or more recent works on stochastic samplers? Is it possible to integrate the solver in these stochastic samplers, as these samplers seem to provide significantly better sample quality compared to their deterministic counterparts on complex datasets? [1] Fast Sampling of Diffusion Models with Exponential Integrator, Zhang et al., ICML 2023 [2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et al., NeurIPS 2022 [3] Gotta go fast when generating data with score-based models. A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas. CoRR, abs/2105.14080, 2021.

Questions

- The author mentioned that EMS computed on the model without guidance can work for guided sampling cases within a common range of guidance scales. I'm wondering how large is the performance gap between algorithms using EMS computed on the model with guidance and the model without guidance. How is this affected by different magnitudes of guidance scales? - Theoretically and empirically, is the model robust to estimation error of EMS? - How is DPM-Solver-v3's performance affected by single-step versus multistep sampling, in comparison with the baselines? - The estimated $s_{\lambda}$ seems to be unstable w.r.t $\lambda$ in some cases. Would using moving average or other smoothing methods help with the sampling algorithm? - I think it would be very helpful to compare the FID score / CLIP score on Stable-Diffusion since low MSE does not necessarily imply better sample quality (the quality is affected not only by discretization error but also by estimation error of neural networks). - How well is DPM-Solver-v3 compatible with current SDE samplers?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

yes

Summary

This work proposes a method to speed up the DPM solver by using empirical model statistics (pre-trained model) as well as employing multistep methods with a predictor-corrector framework for improving sample quality. Promising results were reported.

Strengths

- This work addresses an important problem of fast sampling in diffusion models without retraining. - The proposed parametric model is simple and seems neat with optimality criteria.

Weaknesses

- The improvement of the proposed method over prior works such as DPM-solver++ seems incremental. It is unclear if the proposed empirical model statistics is good enough. While this work discusses an optimality under this parametric model, it is unclear if it is truly optimal in terms of modeling itself. This work may also discuss the prior work such as [H Zheng et al., Fast Sampling of Diffusion Models via Operator Learning, ICML 2023] and its arXiv version, which was able to achieve incredible results with only a single model evaluation. - The experiment results seem to need more improvements and should ensure fairness. For example, Figures 3, 4 indicate that the proposed method is faster than other methods in early iterations, but in order to ensure reasonable FID, one may need a certain number of NFE and in those cases, the differences of FIDs look negligible. Moreover, the comparisons in Fig 6 seems unfair - will other methods achieve similar image quality and FID scores with 15-30% more evaluations? Incorporating the factor of computation should be incorporated for fair comparisons.

Questions

Please address the comments in the Weaknesses section. - Will the speed-up of 15-30% have a practical value? Please discuss.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This work proposes DPM-Solver-v3, an ODE solver for diffusion model inference. The method builds upon existing formulations involving exponential integrators, where the linear term of the ODE is canceled and only the noise predictor needs to be approximated. A new model parameterization is derived which involves an optimal set of coefficients, empirical model statistics (EMS). The authors describe methods for computing the EMS in practice, as well as a pseudo high-order method in the few NFE regime and a half-corrector technique when the guidance scale in guided sampling is large.

Strengths

- Although the method builds upon existing methods involving exponential integrators in the diffusion literature, the model parameterization involving EMS is novel. - Experiments and ablations are thorough and clearly done. Results are compelling and the proposed method consistently outperforms existing ODE solvers across datasets/models, especially in few NFE regime. - Theoretical results for local/global error are clearly presented and assumptions seem reasonable.

Weaknesses

- In the experiments, the authors mention that "we tune the strategies of whether to use pseudo-order predictor/corrector at each NFE on CIFAR10" (lines 254-255). This seems to be an important caveat and should be reiterated in the appendix where the detailed empirical results are presented (Table 4). - Similarly, for Stable Diffusion, s=7.5, the authors mention that the best results among no/half/full corrector are reported. This should be reiterated in the appendix in Table 7. - In general, I wonder if the presentation of the paper would be more compelling if some of the quantitative results could be moved to the main body from the appendix.

Questions

- FID degenerating from 12.76 to 15.91 for ScoreSDE on CIFAR10 (line 822, G.3) in whether or not to use pseudo-order predictor/corrector seems quite substantial. In which cases is the pseudo-order predictor necessary? It might be compelling to produce a table similar to Table 10 (pseudo-order corrector experiments) examining the influence of the pseudo-order predictor. - How does choice of N, K for the EMS change as the timestep scheduler changes?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately address the limitations of the work.

Summary

This paper proposed an improved/generalized version of DPM-solver. It include three more coefficients based on the semi-linear ODE solution proposed by DPM-solver, accounting for minimizing the 'linearity' for the nonlinear component of the ODE and minimizing the first-order discretization error respectively. Such optimal coefficients can be approximately calculated by Monte Carlo samples from the learned model. Together with multistep formulation, predictor-corrector framework by UniPC, and some practical techniques, the paper empirically demonstrated the proposed sample can outperform existing ones especially when NFE is very small.

Strengths

- The paper is well-written and easy to follow. - The paper built on DPM-solve formulation and generalized it with additional coefficients that are optimized based on trained model. The added coefficients are intuitive. Theoretical analysis has been carried out to ensure local/global convergence. - Two practical techniques has been proposed and seems to be useful in general. - Extensive experiments have been performed to justify the effectiveness.

Weaknesses

- I'm not convinced by optimizing $s_\tau$ with Eq. (11): - In equation 10, the first coefficient inside the big integration should be $e^{\int_{\lambda_s}^\lambda (l_\tau + s_\tau) d\tau}$ instead of $e^{\int_{\lambda_s}^\lambda l_\tau d\tau}$. Therefore, the first-order discretization error not only depends on $f_\theta^{(1)} - s_\lambda f_\theta - b_\lambda$. There could be a chance that Eq. (11) is minimized but this term $e^{\int_{\lambda_s}^\lambda (l_\tau + s_\tau) d\tau}$ is amplified. - Main results shown in this paper are not really by first-order solver but by higher-order solvers. A justification of why minimizing first-order discretization error can help higher-order solvers is needed. - A downside of the introduced coefficients is that the integrals in Eq. (14) become intractable, which has to be approximated by trapezoidal rule. It could introduce extra approximation error and add computational cost. I'd like to see a detailed analysis on the extra error and computational cost. - It is understandable and acceptable that this work is heavily built on DPM-solver, but certain sentences in this paper is duplicated from DPM solver. E.g., line 94 our first key insight is..., line 105 our second key insight is ..., line 135-136: the proposed solvers and analysis ....The paper should at least rephrase these sentences. - The paper can do a better job on clarifying which is the unique contribution of this paper while which part has been proposed by previous works. For example, most techniques and theoretical conclusions in section 3.2 have already been developed in DPM-solver / DPM-solver++ / UniPC, which should be further admitted and clarified.

Questions

- Why EMS without guidance can be directly applied to model with guidance? Does it still hold if the guidance weight is really high? - Why for conditional setting, the best setting is 2nd-solver instead of higher-order counterparts?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

I'd expect to see more discussion on limitations due to the intractable integrals.

Looking forward to further feedback

Dear Reviewer oHv2, We thank you very much again for the great efforts on reviewing our manuscript and providing the valuable comments to further improve. We hope you may find our response (as well as other reviews) satisfactory and increase the rating accordingly. If you have any further feedback, we would be very happy to reply. Best, Authors

Looking forward to further feedback

Dear Reviewer wxE7, We thank you very much again for the great efforts on reviewing our manuscript and providing the valuable comments to further improve. We hope you may find our response satisfactory and increase the rating accordingly. If you have any further feedback, we would be very happy to reply. Best, Authors