Library

SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models

A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise to data, while a model learns to gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, a process which has shown to be prohibitively slow. Numerous efforts on speeding-up this process have consisted on crafting powerful ODE solvers. Despite being quick, such solvers do not usually reach the optimal quality achieved by available slow SDE solvers. Our goal is to propose SDE solvers that reach optimal quality without requiring several hundreds or thousands of NFEs to achieve that goal. We propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS), improving and generalizing Exponential Integrator approaches to the stochastic case on several frameworks. After carefully analyzing the formulation of exact solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear part of such solutions. Inspired by the Exponential Time-Differencing method, SEEDS use a novel treatment of the stochastic components of solutions, enabling the analytical computation of their variance, and contains high-order terms allowing to reach optimal quality sampling $\sim3$-$5\times$ faster than previous SDE methods. We validate our approach on several image generation benchmarks, showing that SEEDS outperform or are competitive with previous SDE solvers. Contrary to the latter, SEEDS are derivative and training free, and we fully prove strong convergence guarantees for them.

cs.LGcs.CVmath.NAGenerative Adversarial Networks and Image SynthesisModel Reduction and Neural NetworksGaussian Processes and Bayesian InferenceComputer ScienceMathematics
Source

Paper

References (26)

01GENIE: Higher-Order Denoising Diffusion SolversTim Dockhorn, Arash Vahdat, Karsten Kreis2022 · Neural Information Processing Systems · 158 citations In Library
02gDDIM: Generalized denoising diffusion implicit modelsQinsheng Zhang, Molei Tao, Yongxin Chen2022 · International Conference on Learning Representations · 160 citations In Library
03DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 StepsCheng Lu, Yuhao Zhou, Fan Bao et al.2022 · Neural Information Processing Systems · 2.5k citations In Library
04Elucidating the Design Space of Diffusion-Based Generative ModelsTero Karras, M. Aittala, Timo Aila et al.2022 · Neural Information Processing Systems · 3.7k citations In Library
05Fast Sampling of Diffusion Models with Exponential IntegratorQinsheng Zhang, Yongxin Chen2022 · International Conference on Learning Representations · 632 citations In Library
06Pseudo Numerical Methods for Diffusion Models on ManifoldsLuping Liu, Yi Ren, Zhijie Lin et al.2022 · International Conference on Learning Representations · 909 citations In Library
07Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic ModelsFan Bao, Chongxuan Li, Jun Zhu et al.2022 · International Conference on Learning Representations · 420 citations In Library
08Score-Based Generative Modeling with Critically-Damped Langevin DiffusionTim Dockhorn, Arash Vahdat, Karsten Kreis2021 · International Conference on Learning Representations · 286 citations In Library
09WaveGrad 2: Iterative Refinement for Text-to-Speech SynthesisNanxin Chen, Yu Zhang, H. Zen et al.2021 · Interspeech · 102 citations In Library
10Score-based Generative Modeling in Latent SpaceArash Vahdat, Karsten Kreis, Jan Kautz2021 · Neural Information Processing Systems · 891 citations In Library
11Gotta Go Fast When Generating Data with Score-Based ModelsAlexia Jolicoeur-Martineau, Ke Li, Remi Piche-Taillefer et al.2021 · arXiv.org · 266 citations In Library
12Grad-TTS: A Diffusion Probabilistic Model for Text-to-SpeechVadim Popov, Ivan Vovk, Vladimir Gogoryan et al.2021 · International Conference on Machine Learning · 760 citations In Library

Scroll for more · 14 remaining

Similar papers

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 StepsCheng Lu, Yuhao Zhou, Fan Bao et al.2022 · Neural Information Processing Systems · 2.5k citations In Library
DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model StatisticsKaiwen Zheng, Cheng Lü, Jianfei Chen et al.2023 · arXiv (Cornell University) · 190 citations In Library
SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion ModelsShuchen Xue, Mingyang Yi, Weijian Luo et al.2023 · arXiv (Cornell University) · 87 citations In Library
A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic ModelsEn-hao Liu, Xuefei Ning, Huazhong Yang et al.2023 · International Conference on Learning Representations · 16 citations In Library
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic ModelsCheng Lu, Y. Zhou, Fan Bao et al.2022 · arXiv (Cornell University) · 1.1k citations In Library
Fast Sampling of Diffusion Models with Exponential IntegratorQinsheng Zhang, Yongxin Chen2022 · International Conference on Learning Representations · 629 citations In Library
Elucidating the solution space of extended reverse-time SDE for diffusion modelsQinpeng Cui, Xinyi Zhang, Bao, Qiqi et al.2023 · arXiv (Cornell University) · 8 citations In Library
Differentiable Solver Search for Fast Diffusion SamplingShuai Wang, Zexian Li, Qipeng Zhang et al.2025 · International Conference on Machine Learning · 7 citations In Library
On Accelerating Diffusion-Based Sampling Process via Improved Integration ApproximationGuoqiang Zhang, Niwa Kenta, W. Kleijn2023 · International Conference on Learning Representations · 10 citations In Library
Accelerating Diffusion Sampling with Optimized Time StepsShuchen Xue, Zhaoqiang Liu, Fei Chen et al.2024 · Computer Vision and Pattern Recognition · 80 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer YvZd6/10 · confidence 4/52023-07-07

Summary

There are powerful ODE solvers to speed up the sampling process of diffusion models. Despite being quick, ODE solvers do not usually reach the optimal quality achieved by SDE solvers which are however slow. To tackle this problem, the paper proposes SEEDS, which is based on Exponential Integrator in the stochastic case. SEEDS are derivative and training-free solvers with proven convergence order.

Strengths

1. The authors provide exact solutions for diffusion SDEs and analytical computation of their variance. 2. The paper contains rigorous theoretical derivation and convergence order guarantee for the proposed methods. It also points out its connection with gDDIM. 3. The paper is overall well-written and self-contained. 4. The method achieves outstanding sampling performance.

Weaknesses

The proposed SEEDS method still needs around a hundred NFEs to achieve good sampling qualities, the quality will degrade fast when NFEs decrease, which limits its applications.

Questions

Would you please include the connection between the proposed method SEEDS-1/2/3 and the stochastic Runge-Kutta method properly combined with Exponential Integrator? Also, the convergence order included is order 1 for SEEDS-1 in the mean-square sense and order 1 for SEEDS-2/3 in the weak sense, since the stochastic Runge-Kutta method can achieve higher convergence order even in the strong sense, the reviewer is concerned the convergence order provided is not optimal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author addressed the limitations of their work.

Reviewer daVP6/10 · confidence 4/52023-07-09

Summary

This paper proposes an off-the-shelf (i.e., no further training) few-step sampler for diffusion probabilistic models (DPMs). By isolating linear terms in the exact solution of diffusion SDEs and using a change-of-variable method, the proposed sampler, SEEDS, simplifies the integrals and can approximate the solution of stochastic equations. SEEDS can accelerate the sampling of DPMs without compromising the quality of the samples.

Strengths

- (Quality & Clarity) The authors have successfully provided a theoretical derivation and error analysis of their proposed method, as well as details of the experiments. The paper is well-written and the reviewer had no difficulty comprehending its contents. - (Significance) Numerical experiments show that SEEDS produces competitive results compared to previous sampling methods in terms of quality, with fewer function evaluations, in several benchmarks, namely CIFAR-10, CelebA-64, and ImageNet-64.

Weaknesses

- (Originality) The proposed method has shown promising performance. However, its theoretical background is based on a classical idea of the *variation-of-parameters* formula in the literature of differential equations. Although the authors claim that equation (5) is a novel representation of diffusion SDEs, this kind of solution form is very well-known in the SDE community even for general Hilbert spaces (e.g. see Curtain & Falb and Da Prato & Zabczyk). Therefore, it seems that justification is needed to claim that the contribution of this paper regarding the solution representation suggested by the author is novel. Furthermore, the derivation of exponentially weighted integral is quite similar to the DPM-solver (Lu et al.), except for utilizing Itō-Taylor expansion due to the diffusion part in the solution representation. The convergence result of the proposed algorithm may not be an incremental result. However, this paper's argument on the contribution has an excessive aspect, and it is quite difficult for reviewers to agree with the author's claim. [Curtain & Falb] Stochastic Differential Equations in Hilbert Space, *Journal of Differential Equations*, (1971). [Da Prato & Zabczyk] Stochastic Equations in Infinite Dimensions, (2014). [Lu et al.] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).

Questions

To discuss the generation speed of the SEEDS-3 algorithm in Table 1, it is necessary to compare it quantitatively with the actual runtime, as well as with NFE. For example, in the case of the DPM-Solver, a comparison of DDIM with respect to runtime was made in Table 7 of Appendix E.7.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors mentioned the limations of their work.

Reviewer MHs83/10 · confidence 3/52023-07-13

Summary

The paper proposed an efficient method for solving diffusion SDEs with some convergence guarantee. Experiments are done to verify their claims.

Strengths

The experiments show SEEDS-3 achieves optimal results with minimal NFE on many data sets.

Weaknesses

1. Authors claim they have found a novel representation of the exact solution of the SDE, page 3, line 106. However, it is just a simple technique that's commonly used when solving SDEs. For example, when solving OU process, you multiply some exponential multiplier like this. I don't find anything novel about this. 2. Essentially, in the solution, the first and last terms are treated using a multiplier and change of variables. The second is rewritten using Taylor expansion. The convergence theorem is sort of too straightforward, as the Taylor expansion of course approximates the original function. 3. The experiments only report the NFE taken when achieving optimal performance, which I am not sure is a commonly accepted metric of evaluation in this community. Please comment on this. I am especially open to modifying the score if this question is addressed sufficiently.

Questions

See weaknesses.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

yes

Reviewer CfgK3/10 · confidence 3/52023-07-31

Summary

This paper proposes a sampling method that achieves 3 to 5 times faster computation by exploiting the semi-linearity in the time-reversal stochastic differential equation (SDE). It analytically computes the linear part and reaches optimal quality sampling. This sampling method demonstrates comparable results to existing SDE solvers on datasets like CIFAR10 or Celeba without requiring any additional training methods.

Strengths

The paper is well-written in a clear and readable manner. It is easy to understand the paper without much difficulty, and the derivation of formulas is well-guided, focusing on the essential parts. One of the strengths of the paper is that it proposes a method that achieves approximately 3 to 5 times faster computation while still reaching optimal performance, which sets it apart from other ODE-based methods. Additionally, the paper provides a thorough and clear analysis of convergence, which is commendable.

Weaknesses

In (5), the claim of a "Novel representation of exact solutions of diffusion SDEs" seems to be overclaimed as the equation is well-known in the form of the weak solution of the SDE. Therefore, it is difficult to consider this as a newly discovered novel representation. Moreover, while this integral representation leads to a practical sampling method, much of it is based on formulas used in the DPM-solver [1]. This suggests that the contribution of this work may be relatively weak. The main difference from the DPM-solver appears to be the inclusion of a stochastic term, but beyond that, significant distinctions are not apparent. The claimed advantage of reducing the number of function evaluations (NFE) by 3 to 5 times seems less compelling when compared to ODE-based methods that can achieve sampling speeds more than 50 times faster and still attain optimal performance. For more persuasive results, it would be better if this method demonstrated similar NFE to ODE-based methods while reaching optimal performance. Additionally, it is unclear whether this method addresses any meaningful issues beyond the problem of ODE-based methods not achieving optimal performance. [1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).

Questions

- Is it possible to maintain fast sampling speeds for datasets with resolutions higher than 256, such as CelebA-HQ or ImageNet? - Apart from the ability to reach optimal performance, are there any other distinguishing features compared to the DPM-solver? For instance, does it generate a more diverse set of samples or exhibit stronger capabilities in tasks like imputation?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

1 poor

Limitations

The paper has effectively summarized its limitations.

Reviewer nfDR6/10 · confidence 4/52023-08-01

Summary

This paper aims to accelerate the SDE solvers through exponential integrators. The authors compute the linear part of the SDE analytically, employ a more accurate interaction of the stochastic part. Experimentally, the proposed method improves over the previous ODE solvers.

Strengths

- The paper proposes to utilize the exponential integrator to solve the diffusion SDEs (Eq. (5)). In addition, it approximates the integral components through Taylor expansion. - The paper utilizes the stochastic exponential time differencing method to compute the variance of the stochastic term analytically. - Experimentally, the third-order solver yields better FID than previous ODE solvers while requiring significantly smaller NFEs than SDE solvers.

Weaknesses

- It seems that the same formula of the exponential integrated version of SDE (Eq.5) has been proposed in prior works [1] (see their eq.17, arXiv version 1). Since the DPM-solver [2] is essentially the same as DEIS, and they also use a Talyer expansion of the score function, I wonder how this work differs from these prior works. - The authors show in Prop 4.4 that setting $g=0$ in Eq.15, the resulting SEEDs solvers do not equal to DPM-solver. I wonder why this is the case based on the point above. In addition, setting $g=0$ in Eq.15 would not lead to meaningful backward ODE/SDEs. - The SEEDS-3 solver indeed showcases great improvement over ODE solvers. However, when using an increased NFE, the EDM solvers perform better than SEEDS-3 (SEEDS-3 even gets worse when increasing NFE as shown in Fig.1). In addition, I notice the SEEDS solver does not even apply to a low NFE regime (NFE<100) on CIFAR-10. It might be helpful to utilize the techniques in a concurrent work [3] to further improve the sample quality when increasing the NFE, or to achieve better results in the low NFE regime. It also is helpful to discuss [3] as they also focus on improving the existing SDEs. ### Minors - Line 74 : $\mu_t \to \alpha_t$ [1] Zhang, Qinsheng, and Yongxin Chen. "Fast sampling of diffusion models with exponential integrator." arXiv preprint arXiv:2204.13902 (2022). [2] Lu, Cheng, et al. "Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps." Advances in Neural Information Processing Systems 35 (2022): 5775-5787 [3] Xu, Yilun, et al. "Restart Sampling for Improving Generative Processes." arXiv preprint arXiv:2306.14878 (2023).

Questions

- Is this work a straightforward application of prior works ([1], [2]) to SDEs? - In Figure 1, why does the SEEDS solver have a such high FID score (worse sample quality) in the low NFE regime? Is it due to the error when simulating the noise term?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussed in Sec 6.

Reviewer CfgK2023-08-18

Looking at the new change of variable, in the end, it seems the model calculates the score model for one NFE twice. It's uncertain whether the NFE has decreased, but the total running time might not differ significantly. I'm not sure if this can be considered a significant contribution...

Authorsrebuttal2023-08-18

We thank the reviewer for their response, to which we happily bring answers and clarifications. Both in [DPM-Solver] and in our method, the score model is calculated exactly once per NFE (this is just the definition of "Number of Function Evaluations"). Since theoretically both DPM-Solver and SEEDS compute analytically all components in the sampling method, then the total runtime is expected to increase linearly with respect to increasing NFEs in both cases, as was experimentally checked. You may also verify that there is no significant improvement done, in terms of runtime vs NFEs, between DDIM and DPM-Solver in Table 7 of Appendix E.7 in [DPM-Solver]. But DPM-Solver achieves good FID score significantly faster than DDIM, a fact that can't be seen just by looking at runtime vs NFEs. As such, one cannot evaluate the significance of a sampling method solely on analysing the runtime vs NFE behaviour. In our case, since we compute every term (both with the VP or our new EDM-optimized change of variables) analytically, then the computing cost of anything other than the score model's pass is negligible. That being said, the NFEs that are necessary to obtain state-of-the-art FID score in SEEDS shows a significant improvement compared to other SDE sampling methods. It shows to be twice faster than optimized EDM sampling method on ImageNet-64 as was reported in our experiments. We hope that this, and the involved theoretical proofs we made in this work that have no equivalent in the current literature on the subject, convinces you of our contribution.

Reviewer CfgK2023-08-18

Thank you for your quick response. However, I find the author's reply somewhat disappointing. Are you claiming that your paper is acceptable based solely on improvements in NFE, since the DPM-solver evaluation is also done in NFE? During the rebuttal process of DPM-solver, the authors were asked to provide information about runtime, as they initially only mentioned NFE and omitted runtime. While it's true that there are many factors influencing runtime, such as architecture and various perspectives, isn't it true that the linear increase in time during the sampling process is a fact? What is the original purpose of reducing NFE? It seems people wanted to decrease the total sampling time, and one of the indicators they used is NFE. However, isn't it somewhat illogical to argue that even though total time is similar, the reduction in NFE is acceptable? Of course, it's reasonable to assume that performing analytic calculations based on the order in Taylor expansion would increase accuracy and thus improve performance. However, if performing multiple model calculations per NFE leads to a linear slowdown in speed per calculation, wouldn't this diminish the practical merits? Also, there are various methods to apply the change of variables, so it's possible to derive non-equivalent equations. Is this really significant and unique enough to be the main contribution? I still haven't received a satisfactory answer regarding this matter.

Authorsrebuttal2023-08-18

Thank you for your reply. In the hope of making our answer satisfactory, let us go step by step. First, we'd like to stress that the NFEs and the number of steps in a solver are not the same. For instance, DPM-Solver-3 uses 3 NFEs per step and achieves an acceptable FID score with as few as 15 NFEs (meaning 5 steps). > Are you claiming that your paper is acceptable based solely on improvements in NFE, since the DPM-solver evaluation is also done in NFE? We claim, in our general rebuttal, 4 points that we find are worthy novelties in our work. At an experimental level: improvements are done on both FID and NFE, by prioritising FID. The question we try to answer is: "Can we obtain current SOTA FID with fewer NFEs than those that were needed by the sampling method that achieved that SOTA FID ?". Notice that this is not the same as DPM-Solver, who prioritise NFEs. Their question is, "At fixed low NFEs, can we get a SOTA FID, even if this FID will not become the absolute SOTA FID found for that model (regardless of how many NFEs were used to achieve it)? > During the rebuttal process of DPM-solver, the authors were asked to provide information about runtime, as they initially only mentioned NFE and omitted runtime. While it's true that there are many factors influencing runtime, such as architecture and various perspectives, isn't it true that the linear increase in time during the sampling process is a fact? We included the runtime vs. NFE behaviour in our rebuttal to reviewer daVP. It might seem like a fact, but derivative-based methods will need to do score model's computation and also compute their derivatives. This might break such linearity. A "sanity check" we experimentally ensured that nothing unexpected might be happening. As SEEDS is fully derivative-free, we checked such expected linearity. > What is the original purpose of reducing NFE? Talking about NFEs arose when solvers using multiple NFEs per step started to appear in the DPM literature. If one only thinks about the number of sampling steps, then one would find that DPM-Solver-3 only needs 5 steps to achieve average sampling quality. As a first thought, one would think that this means that they need to make 5 score model's inference to do so, but it actually uses 15 score model's inferences. As such, reducing the NFEs to reach a targeted FID (either optimal FID like SEEDS, or average FID like DPM-Solver) is a more pertinent evaluation metric for speeding up sampling than the number of steps. > However, isn't it somewhat illogical to argue that even though total time is similar, the reduction in NFE is acceptable? Notice that when comparing runtime and NFEs, what really matters is the behaviour of the increase of runtime when increasing NFEs. Experimenting on it somehow is a sanity check that your sampling algorithm is doing what we expect from it. > However, if performing multiple model calculations per NFE leads to a linear slowdown in speed per calculation, wouldn't this diminish the practical merits? Thank you for this question that allows us to make a good example to clarify the roles of steps and NFEs. On the contrary, multiple model calculations (which are called NFEs) per sampling step (and not per NFE) actually have shown to increase the practical merits. A method that uses 1 NFE per step (like DDIM) but needs, say, 50 steps (so 50 NFEs) to get average FID, will be much slower than a method (like DPM-Solver) that uses 3 NFEs per step but only needs 5 steps (so only 15 NFEs) to get that same average FID. > Also, there are various methods to apply the change of variables, so it's possible to derive non-equivalent equations. Is this really significant and unique enough to be the main contribution? As we responded in the general rebuttal response that you may find at the beginning of this webpage, we detailed what we claim are our principal contributions. Besides our EDM change of variables, we cited as main novelties - the use of the Stochastic Exponential Time-Differencing method (not incremental over DPM-Solver), - the use of a specially chosen Itô-Taylor expansion (that is very different from its deterministic Taylor counterpart) - and mainly the Markov-chain preserving choice of multiple analytic stochastic noise contributions for each higher-order correction in each step. It is, in particular, this last point that we believe is the most novel as it has no stochastic counterpart to our knowledge. Notice that DPM-Solver is heavily inspired by tables 5.3, 5.4, and 5.8 in [HO] to obtain their solvers. Our choice of stochastic components has, to our knowledge, no precedent in the Numerical Analysis literature. Taken as a whole and backed by experiments, we consider this set to be unique and significant, and we are committed to keep the dialog open, hoping it would be deemed satisfactory enough for your endorsement. [HO] Hochbruck, Ostermann. Explicit Exponential Runge-Kutta Methods for Semilinear Parabolic Problems

Reviewer CfgK2023-08-18

Thank you for your response. However, ultimately, I decided to lower the rating. Firstly, the equation (4) introduced for the stochastic Exponential Time-differencing method is not new, but a well-known formula. Secondly, the Itô-Taylor expansion introduced to derive the sampling is fundamentally not different from the deterministic Taylor expansion. This is because the noise term for equation (4) can be directly calculated using the Itô isometry, and practically, "the deterministic Taylor expansion" is used for the drift term. The technique used to expand (6) is already employed in the DPM-solver. It's hard to consider obtaining a different formula from the DPM-solver by setting a different change of variables as a significant distinction. Moreover, it's also challenging to perceive significant advantages from an order-of-magnitude perspective. The claimed advantage, apart from reducing NFE, is the improved robustness, but the specific benefits in terms of generation aspects and why approximating with the addition of stochastic terms are better are not explained. The contribution aspect appears to be lacking, and I still feel that I haven't received a satisfying answer to my question.

Authorsrebuttal2023-08-19

We would like to point out to the reviewer that: - SETD is not the same as the exponential representation of the exact solution (eq.4). We already clarified eq. 4, and the SETD method means expressing a stochastic integral in terms of $\varphi$-functions (and you need exponentially weighted stochastic integrals to be able to do so here). - The explicit form of the Itô-Taylor expansion (see App. D.2) to degree 2 is needed to show that SEEDS-1 is not of strong order 0.5 but 1.0: SEEDS cannot be theoretically grounded by a purely deterministic Taylor-like argument. - One can put a highly precise sampling method to a badly trained DPM that the generation process will still be bad. This is because the sampling error and the learning approximation error are independent one to the other. - Itô Isometry is not sufficient to successfully determine the stochastic contributions in SEEDS. In addition to the ablation experiment done in Fig. 1 of the appended PDF, the theoretical explanation of our choice of the stochastic components is contained in Appendix B.2.4 and B.3 in an implicit for. We have singled it out below to make a clear explanation of it and will be added to our manuscript to the request of the reviewer. This also clarifies how changing the choice in the stochastic terms leads inevitably to failing to satisfy independence of brownian increments on non overlapping intervals and generalises to SEEDS-3. Hopefully this will convince you that our approach is theoretically grounded. **The sequence $(\tilde{\mathbf{𝐱}}_t )_t$ induced by the choice of stochastic noise contributions in SEEDS-2 presented in Alg. 3 satisfies the Markov property.** **Proof**: On the one hand, we have $$\mathbf{u}_1=\frac{\alpha\_{s_1}}{\alpha_s} \tilde{\mathbf{x}}_s-2\sigma\_{s_1} \left( e^{\frac{h}{2}} - 1 \right) \mathbf{\epsilon}\_{\theta} (\tilde{\mathbf{x}}_s, s) - B^1, \qquad B^1:=\sqrt{2}\alpha\_{s_1} \int\_{\lambda\_s}\^{\lambda\_{s_1}} e^{- \lambda} d \omega\_{\lambda}.$$ Now $B^1$ depends on the Brownian movement $\left( \omega\_{\lambda\_{s_1}}-\omega\_{\lambda\_s} \right)\_{\lambda\_{s_1} \geq \lambda_s}$. By the Markov property, this is independent of $(\omega\_{\lambda_u})\_{\lambda_u \leq \lambda_s}$. Since $\tilde{\mathbf{x}}\_s$ is a function of $(\omega\_{u} )\_{u \geq s}$, we deduce that $B^1$ has to be independent of all $( \tilde{\mathbf{x}}\_u )\_{u \geq s}$. By Itô Isometry, we obtain (noticing that $h\varphi(h)=e^h - 1$ to make clear the use of the SETD method in our approach) $$B^1 = \sigma\_{s_1} \sqrt{e^h - 1} z^1, \quad z^1 \sim \mathcal{N} (\mathbf{0}, \mathbf{I}\_d).$$ On the other hand, the update $\tilde{\mathbf{x}}\_t$ is $$\tilde{\mathbf{x}}\_t = \frac{\alpha_t}{\alpha_s} \tilde{\mathbf{x}}\_s - 2 \sigma_t (e^h - 1) \mathbf{\epsilon}\_{\theta} (\tilde{\mathbf{x}}\_{s_1}, s_1) - B^0, \qquad B^0 = \sqrt{2} \alpha_t \int\_{\lambda\_s}\^{\lambda\_t} e^{- \lambda} d \omega\_{\lambda}. $$ Then, by using independent increments property of Wiener process $ \{ \omega_{\lambda} : \lambda \in [\lambda_s, \lambda_t] \}$, we can deduce, for $0 \leq \lambda_s < \lambda_{s_1} < \lambda_t$, that $$ \omega\_{\lambda\_{s_1}} - \omega\_{\lambda_s} \quad \text{and} \quad \omega\_{\lambda_t} - \omega\_{\lambda\_{s_1}}$$ are independent. Then, by the above Brownian independence property, the random variables $B^0$ and $B^1$ are - Independent on non-overlapping time intervals - Dependent on overlapping intervals. By Chasles rule, we then decompose $$ B^0 = \sqrt{2} \alpha_t \int\_{\lambda\_s}\^{\lambda\_{s_1}} e^{- \lambda} d \omega\_{\lambda} + \sqrt{2} \alpha_t \int\_{\lambda\_{s_1}}\^{\lambda\_t} e^{- \lambda} d \omega\_{\lambda} $$ and, as $$\int\_{\lambda\_s}^{\lambda\_{s_1}} e^{- \lambda} d \omega\_{\lambda} = \frac{B^1}{\sqrt{2} \alpha\_{s_1}}, $$ we obtain $$B^0 = \sqrt{2} \alpha_t \left[ \frac{B^1}{\sqrt{2} \alpha\_{s_1}} + \int\_{\lambda\_{s_1}}\^{\lambda\_t} e^{- \lambda} d \omega\_{\lambda} \right] . $$ Then we have $$B=\sqrt{2} \alpha_t \int\_{\lambda\_s}\^{\lambda\_{s_1}} e^{- \lambda} d \omega\_{\lambda} + \sqrt{2} \alpha_t \int\_{\lambda\_s}\^{\lambda\_t} e^{- \lambda} d \omega\_{\lambda}= \sqrt{2} \alpha_t \frac{1}{\sqrt{2}} \frac{\sigma\_{s_1}}{\alpha\_{s_1}} \sqrt{e\^{2 (\lambda\_{s_1} - \lambda\_s)} - 1} z^1 + \sqrt{2} \alpha_t \frac{1}{\sqrt{2}} \frac{\sigma_t}{\alpha_t} \sqrt{e^{2 (\lambda_t - \lambda_s)} - 1} z^2 = \frac{ \sigma\_t}{\alpha_t} \left( \frac{\sigma\_{s_1} \alpha_t}{\sigma_t \alpha\_{s_1}} \sqrt{e^h - 1} z^1 + \sqrt{e^{2 h} - 1} z^2 \right) = \sigma_t \left( e\^{\lambda_t - \lambda\_{s_1}} \sqrt{e^h - 1} z^1 + \sqrt{e^{2 h} - 1} z^2 \right) = \sigma\_t \left( e^{\frac{h}{2}} \sqrt{e^h - 1} z^1 + \sqrt{e^{2 h} - 1} z^2 \right) = \sigma\_t \left( e^{\frac{h}{2}} \sqrt{h \varphi_1(h) } z^1 + \sqrt{2h \varphi_1(2h)} z^2 \right) = \sigma\_t \left( \sqrt{e^{2 h} - e^h} z^1 + \sqrt{e^{2 h} - 1}z^2 \right) $$ which completes the proof.

Authorsrebuttal2023-08-21

Hoping to fully address the reviewer's concerns satisfyingly, we would like to make some further points that might have been missed in our initial responses. > The reviewer asked about the extent to which our work potentially addresses distinguishing features compared to DPM-Solver beyond optimal performance, like sample diversity or exhibiting stronger capabilities in tasks like imputation. We proposed the distinguishing features of SDE solvers for robustness tasks in comparison to ODE solvers. As a great number of DPM-based papers have been widely adopted in the field of robustness, which is very significant in ML, we hope this to be a first acceptable answer. > As we understand, the reviewer's concern is centred on the significance of our contribution and, in particular, the extend to which it differs from [DPM-Solver]. The reviewer pointed out the similarities between these two works: the use of the variation-of-parameters formula and the use of a change of variables to obtain exponentially weighted integrals, the similarity between the Taylor expansion of [DPM-Solver] and our truncated Itô-Taylor expansion. In l.42 of our introduction we stated that one of the objectives of SEEDS is to enhance and generalise [DPM-Solver] to the stochastic case in various frameworks (like the VE, iDDPM and EDM frameworks in Tab.1 in [EDM]) so these similarities are to be expected. The contribution we would like the reviewer to focus on is Theorem 4.1. The involved proof of this result shows the extent to which generalising DPM-Solver to the stochastic case is not trivial precisely from the point of view of **showing strong order convergence guarantees**. - Heuristically **constructing SEEDS** has, as distinctive novelty, the principled choice of noise components whose theoretical justification was clarified during this rebuttal. - The EDM change of variables allowed our main **experimental feature**: to match the EDM solver's performance on ImageNet64 twice faster, and without needing any optimisation procedure to do so. - Formally, we believe that **proving that SEEDS are SDE solvers with convergence guarantees** should be seen as a distinctive and significant contribution. Up to our knowledge, **none of the proposed SDE solvers for DPMs have provided such guarantees**. Actually [EDM, Sec. 4 & Tab. 5] states that, in their sampling method with injected stochasticity, their optimal choices for $S_{churn},... , S_{noise}$ all break convergence guarantees (due to the choice $S_{noise} > 1$). So it cannot be seen as a proper (well-conditioned) SDE solver. Convergence guarantees are what rigorously justify that a sampling scheme is reliable. For instance, it is shown in [R] that the convergence order for DPM-Solver (in Data Prediction mode) has been over-estimated (i.e. **suffer from order reduction** [H, Sec. 5]) and propose a new exponential integrator based on [H, Tab. 5.9] to enforce correct convergence order degree, consequently showing generation quality improvements. In contrast, our proven convergence orders are optimal, not overestimated, and need the full Itô-Taylor expansion expression to be proven. We hope this may satisfy the reviewer's concern on our contributions. > The specific benefits in terms of generation aspects and why approximating with the addition of stochastic terms are better are not explained We provided explanation of the stochastic terms. Benefits of SEEDS, as a SDE solver, in terms of generation aspects are: - As proven in [L, App. B], even if the score model has been trained to the optimal score function, the distributions between SDEs and ODEs are **still different**. This is because the distribution at time $T$ in the forward process is always not exactly a standard normal distribution as in the reverse SDE/ODE process, and thus the distributions at time 0 are different: **SDE solvers and ODE solvers may have different sample quality, even if the score models are optimal**. - [X, Th.1] shows how, in the large NFE regime, SDE methods (with proven convergence orders) **will outperform ODE methods** in terms of sampling quality. - Both [X, C] show that the sampling error divides into **discretisation** and **approximation** errors, indicating that one can hardly expect from any sampling method to enable DPMs to exhibit significant improvements on generative tasks that the pretrained score model didn't already possess. We kindly ask all reviewers to take this answers into account. [R] Zhang et al. Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling [H] Hochbruck, Ostermann. Explicit Exponential Runge-Kutta Methods for Semilinear Parabolic Problems [L] Lu et al. Maximum likelihood training for score-based diffusion ODEs by high order denoising score matching [X] Xu et al. Restart Sampling for Improving Generative Processes [C] Chen et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Reviewer daVP2023-08-20

Thank you for the detailed response and clarifications. The response sufficiently addresses my questions. After carefully considering the responses, I will be keeping my current rating. I think the authors need to address the remaining concerns of other reviewers sufficiently.

Authorsrebuttal2023-08-20

We warmly thank the reviewer for their response. Following your advice, we will proceed to address the remaining concerns of others reviewers to the best of our effort, hoping it will eventually be considered to be sufficient.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC