Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations
The diffusion model has shown remarkable success in computer vision, but it remains unclear whether the ODE-based probability flow or the SDE-based diffusion model is more superior and under what circumstances. Comparing the two is challenging due to dependencies on data distributions, score training, and other numerical issues. In this paper, we study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. We first introduce a pulse-shape error to perturb the score function and analyze error accumulation of sampling quality, followed by a thorough analysis for generalization to arbitrary error. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient. However, when the perturbation occurs earlier, the SDE model outperforms the ODE model, and we demonstrate that the error of sample generation due to such a pulse-shape perturbation is exponentially suppressed as the diffusion term's magnitude increases to infinity. Numerical validation of this phenomenon is provided using Gaussian, Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10.
Paper
Similar papers
Peer review
Summary
The paper studies diffusion models, which comprise a class of generative models based on stochastic differential equations. In diffusion models, data is first transformed into Gaussian noise via a stochastic differential equation (usually an Ornstein-Uhlenbeck process), and then a backward process, which turns noise into synthetic data, is learned via the score function. In most works, the diffusion coefficient of the backward process is either chosen to be zero, or to be a constant matching the diffusion term of the forward. The first case is an ODE implementation, while the second case is an SDE corresponding to the exact time reversal of the forward process. However, any function of time can be chosen for the diffusion coefficient of the backward process, and in this work the authors explore the consequences of different choices of diffusion coefficient. In particular, they ask how the optimal choice of diffusion coefficient depends on the error induced by the estimated score function. The main results of the paper are as follows. Let $h$ denote the diffusion coefficient of the backward process. Assume that the score function estimator has the for $s_t(x)= \nabla \log p_t(x) + \epsilon E_t(x)$, where the first term is the exact score function and the function $E_t(x)$ is a bounded error function. Let $\hat{q}_T$ denote the law of the backward process at time $T$ with $s_t(x)$ in the place of the true score function. Then under certain assumptions on $p_0$, it holds that $$ KL(p_0 \parallel \hat{q}_T = L(h) \epsilon^2 + O(\epsilon^3),$$ where the leading order term $L(h)$ depends on $h$, $p_0$, $E_t(x)$ and $T$. The leading order term $L(h)$ is computed explicitly, which allows the authors to analyze several specific cases when $h$ is constant. In particular, they show that if the error $E_t(x)$ is time-localized (i.e., only nonzero at $t=s$ for some $s > 0$) in the middle of the time interval, the leading order term $L(h)$ decays exponentially fast to zero as $h \rightarrow \infty$. This suggests that the SDE implementation can significantly outperform the ODE implementation in some cases. They also show that for certain forms of the error function, the opposite is true: the ODE implementation has better KL divergence error than the SDE implementation. -------------- After rebuttal: Thank the authors for the clarifications and comments, which addressed my earlier concerns. I will keep my rating unchanged.
Strengths
Overall, the paper presents a novel and interesting question with some practical implications. The question of choosing the diffusion coefficient in the backward SDE seems largely unstudied because, as the authors mention, the ODE and reverse-time SDE approaches are standard conventions. The fact that they show that different choices of the diffusion coefficient can lead to vastly different performances suggests that the problem is worth studying further. The authors provide a nice explicit characterization of the leading order term in the KL divergence error, the derivation of which requires a lot of work. Namely, they show that as $h \rightarrow \infty$, the term $L(h)$ converges exponentially fast to a constant $\Tau$ which is upper bounded by $$ \Tau \lesssim \frac{1}{2m_0^2} \int_{\mathbb{R}^d} \frac{(\nabla \cdot (p_0 E_t))^2}{p_0} dx.$$ Here, $m_0$ is the second moment of $p_0$. This suggests the quantity on the right-hand side above plays a significant role in the error analysis of diffusion models, and thus it could be insightful to understand it for different classes of probability distributions. The authors also leave a lot of room for interesting future work. Most interestingly, the authors pose the question of how to infer the optimal choice of $h$ from data, which could lead to a lot of subsequent research.
Weaknesses
One weakness of the paper is that the upper bound on the constant $\Tau$ is not further explained through discussion or examples. It seems like the upper bound plays a big role in bounding the KL divergence error for diffusion models. It would be very interesting to discuss explicit examples of distributions $p_0$ where the upper bound on $\Tau$ can be explicitly computed or controlled, other than Gaussian distributions. This could be mentioned as a direction for future work. It would also be helpful to describe intuitively what the bound represents about a distribution (what does a large/small $\Tau$ say about a distribution?). In addition, the numerical experiment on the 2D 4-mode Gaussian mixture could benefit from additional explanation. It doesn’t seem like a machine learning problem, since you already know the score of the distribution from which you are trying to sample. Perhaps the point is simply to illustrate the convergence of $L(h)$ to a constant as $h \rightarrow \infty$, in which case the purpose of the experiment should be stated more clearly.
Questions
1. Can you explain in what setting, if any, one would ever encounter a score estimator where the error term was localized in time? Or was the purpose of this choice of error term simply to illustrate that different choices of diffusion coefficient can lead to very different performance? It seems like a very strong assumption that would never be realized in practice. 2. It would be good to state in the main result any dependence on the sub-Gaussian constant $c_U$. As it is stated in the paper, one often works with a mollified version of the empirical distribution, which leads to a tradeoff between bias from smoothing and faster convergence for learning smoother distributions. Thus, rather than treat the constant $c_U$ as hidden, it could be beneficial to discuss how it balances with other terms. 3. In this paper, only constant choices of the diffusion term $h$ are discussed. Is this just for simplicity? Could it be of interest to consider certain parameterizations of nonconstant functions for $h$? If so, this could be mentioned for future work.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
2 fair
Contribution
2 fair
Limitations
None
Summary
In this work the authors try to understand the impact of noise in the reverse process of diffusion models in the presence of an approximate score network. In particular, they look at how the Kullback-Leibler divergence between the true data distribution and the denoised distribution evolves w.r.t. the diffusion term $h$ of the backward (generative) process, with an asymptotic expansion of this KL divergence and studying the first non-zero term. They then show that if the score has a pulse (delta) error early on in the generative sampling, higher values of $h$ are able to recover the density trajectory better, whilst with and error in the late stage of the denoising process it's the opposite. They empirically back these results on synthetic experiments (With heuristic errors) and on the swiss-roll and MNIST dataset with trained score network.
Strengths
- I believe this paper tackle a really important question, which is the role of noise in the generative process. In particular with the success of flow matching which shows that continuous normalising flows, which are (deterministic) ODE based generative models, can be trained akin to denoising score matching models. - The analysis seems sounds and rigorous, and the assumptions reasonable. - The take home message (if I understood correctly) is that apart if the score error is concentrated at the end, the higher the level of noise the better (with infinite computational budget to actually discretise the backward process). - They link they theoretical findings with empirical results, both on simple settings which is nice to unambiguously be able to conclude, and more realistic settings when the score network is trained.
Weaknesses
- No major weakness, but some of the interpretation of the theoretical results could perhaps be enhanced (cf following questions). - The fact that higher values of $h$ requires more discretisation steps is something that should be further stressed I believe.
Questions
- Eq 5: missing the score? Or this is the process of the error? - Section 3.3: typo in title 'Placn' - Equation 9: This does not assuming $h \gg 0$ right? Worth moving it above in Section 3.2 to avoid any ambiguity perhaps? - line 194: 'By convection [...]' what does this mean? Doesn't this simply come from the definition of the time-reversal? - line 196-200: So when $h \rightarrow \infty $ not only $V_t \rightarrow U_t$ but because of the prefactor $h_t^2/2$ in Eq 9 (left) the distribution $\mu_t$ converges extremely fast to to $\mu_0$? Not sure to know what '“almost quasi-static” thermodynamics' refers to. Likely worth expanding a bit more on this in the main paper? - Can this be seen trough the perspective of Langevin dynamics as a corrector, which would 'project' $\rho_t$ back to $p_t$ at $t>s$? - line 233: '$L(h)$ will decay to zero exponentially fast' with $h$? - Section 3.5: Here the setting in that the error accumulates over time in contract with the 'pulse' of Section 3.4. The results of Section 3.5 are that with error at the end of the generative process the smaller $h$ the better, yet if the pulse is also near the end, wouldn't large $h$ help (as the larger $h$ the faster it would correct this er§ror?). - MNIST: This section is not really self-contained as the weighting functions are in the appendix. What is the motivation for this cubic 'noise' weighting? - appendix: 'The computational cost of SDE-based models with large diffusion (h > g) is relatively high due to the necessity of a larger number of time steps.' What is the reason for this? That is quite key and worthy of being further developed (in the main text).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
- Figure 4: It shows that indeed higher values of $h^2/g^2$ can lead to smaller error yet it also requires a lot more step. At $10^3$ steps, which is quite a reasonable number of steps already, the probability flow ODE ($h=0$) perhaps best and the curves starts overlapping around ~2000-3000 steps. So although the theoretical conclusion is that larger values of $h^2/g^2$ is better, for practical implementations there is a trade-off between larger number of discretisation steps (with larger values of $h$) and larger score network architecture. - The authors look at different types of errors on the score network, both theoretically with pulse-shape error to simplify the analysis, but also experimentally with simple errors (weighting the true score for some time range) which makes it easier to interpret. It would be interesting to look at the empirical error of trained score network (e.g. on image dataset), depending on the weighting $w(t)$ in the loss, but also the scheduling $g(t)$, and in particular its (average) evolution through the denoising time. Is the error spread uniformly (wrt time)? Writing the score as $\epsilon / \sigma(t)$ (with $\sigma(t)$ the marginal variance at time $t$) couldn't one argue that with uniform error in the estimation of noise $epsilon$, the error in the score will blow up near $t=0$ thus the setting where the error is large close to the end of the generative process is the most likely setting (and therefore $h=0$ should work best)? Perhaps this is related with the fact that practitioners stop the denoising process at time $t=\eta$ with $\eta$ a small hyperparameter (e.g. \eta=e-3)? - Would suggest increasing the fontsize of Figures (e.g. Fig 4)
Summary
The authors focus on reverse diffusion process in the presence of non-negligible error in the score function, and estimate KL divergence between the data distribution and the distribution generated by reverse process. They analyze how the KL divergence varies with the diffusion coefficient and demonstrate that a large diffusion coefficient is beneficial when the error of score function is concentrated near beginning of generation process, while a small diffusion coefficient is beneficial in the opposite case. The findings are validated through numerical experiments on Gaussian mixture model and MNIST data set.
Strengths
The paper introduces a novel analysis of how error accumulation occurs in diffusion models when the error in the score function is not negligible. This is in contrast to conventional analysis that bound error between target and generated distributions when the error in the score function is small. The analysis is only employed in exploring the diffusion coefficient in reverse generative process. However, the analysis has potential implications for various applications where the correction or modification of score function is important. For instance, more theoretical treatments could be possible on image editing(classifier-free-guidance), fair generation(discriminator-guidance), or so on.
Weaknesses
In Line 178, relation between the assumption and low-dimensionality of data distribution require further explanation to enhance understanding.
Questions
Regarding the scaling hyperparameter in the loss function of conventional diffusion model, which aims to balance the error at different time of the generative process, can the practical choice of the scaling be associated with the theoretical results presented in this work?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The authors acknowledge limitation of their work. One additional limitation is lack of large-scale experiments to further validate the findings.
I have chosen to maintain my original evaluation. While I acknowledge the valid concerns by reviewer LAed, I believe this work offers a valuable theoretical contribution. I suggest that the authors include a discussion with reviewer LAed to address their concerns and acknowledge the limitation of this work. Additional suggestions: - I do not think CIFAR10 alone is large scale experiments. Therefore, I strongly urge the authors to include more experiments in the revised version. Even if some of the experimental results do not align perfectly with the theory, they would not diminish the value of this work and could instead serve as limitations that could open up future research directions. - To enhance the accessibility of your paper to a wider range of readers, consider adding intuitive figures that visually elucidate the implications of your theorems.
Summary
The paper provides a theoretical analysis of the estimation error of SDE and ODE methods along with some numerical experiments.
Strengths
By perturbing the score function, the authors study how the estimation error changes in ODE and SDE methods.
Weaknesses
It is commonly known that the sample generation error consists of three parts: discretization error, estimation error, and initialization error, see e.g. [1, 2]. It is also commonly known that usually the ODE works better with fewer NFE and SDE works better with more NFE due to the interplay between discretization error and estimation error. The estimation error might not even dominates the total error. I don't think only studying the continuous time model provides very useful insight into how to choose those two methods optimally. There is a significant lack of connection between this theory and what happens in reality. Please comment on this. [1] Chen et al., Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions [2] Lee et al., Convergence of score-based generative modeling for general data distributions
Questions
see weaknesses.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
1 poor
Limitations
yes
Summary
The paper explores the difference between ODE-based probability flow and SDE-based diffusion models when score training errors are present. Specifically, they investigate how setting the generative diffusion coefficient h impacts sample quality.
Strengths
- The question the authors are trying to answer seems to be a good research question. - It's nice to see experiments validating some of their assumptions (Fig 1.).
Weaknesses
- As is, the paper is very difficult to follow. - Many equations are presented, but few of them are explained. - With so many variables presented in the paper, it is difficult to keep track of all of them, making the paper challenging to comprehend.
Questions
N/A
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
- The authors briefly address potential negative societal impacts of their work. - I did not see any mention of limitations of their analysis.
I understand the authors are trying to justify why studying the continuous time model is necessary. However, the reasons are not convincing. 1. "(1) absence of score estimate error, addressed in numerous numerical analysis textbooks; (2) absence of discretization error, requiring an exploration of continuous-time models" The authors seem to suggest that it is common to assume one error is absent and study another one solely. That might be the case in numerical analysis textbooks, but it is not what people do at conferences like this, as far as I know. I'd be really interested to see some references that solely study one of the three errors: score function estimation error, discretization error, or initialization error. In fact, for example, [1] does not neglect the score function estimation error. 2. "The perception of discretization error being uncontrollable perhaps has evolved" My point was not the discretization error being uncontrollable. What I was trying to say was there was no evidence suggesting the discretization error is negligible, compared to the score function estimation error. For example, let's assume the total error is $E=A+B+C$, where $A$ the is score function estimation error, $B$ is the discretization error, and $C$ is the initialization error. I agree with the authors that $B$ is bounded under certain conditions, but this doesn't mean $B\leq A$. To make this study useful, we need some results like $B\ll A$. 3. I think it might be more appropriate to compare this work with, for example, [2], instead of Song's work at ICLR. In [2], they provide a unified framework for flow models and diffusion models using continuous time stochastic processes. Notably, the framework they proposed is new. Given the high similarities in both techniques and results between those two works, and considering this work is only a small subset of [2], I decided to keep the rating for now. I am glad to have more discussions if there are still misunderstandings or confusion. [1] Chen et al., Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions, ICLR, 2023. [2] Albergo et al., Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, 2023
### About comment #1 and #2 These two comments are again about whether a continuous-time model without directly working on the discrete-time setting is still of sufficient scientific contribution or not. In fact, the work of (Albergo et al., Stochastic Interpolants 2023) that the reviewer just appraised in the feedback is entirely based on the continuous-time setting. We believe that it is more beneficial if focusing on the *contributions* of the continuous-time model, especially when the conclusions are both rigorous in theory and also backed up by substantial numerical examples, as we did in our original submission and the new supplementary PDF. Additionally, considering feedbacks from other reviewers, we maintain that utilizing a continuous-time model isn't inherently a defect in our paper. ### About comment #3 Thanks for presenting this argument that did not appear in the initial review report. We are glad to respond and make clarifications, despite that this comment is a bit confusing for us to comprehend. Please allow us to explain based on our understanding. *“Given the high similarities in both techniques and results between those two works, and considering this work is only a small subset of [2]”* + “the high similarities in both techniques and results between those two works”. We are quite puzzled by why those two works from others (Albergo et al., Stochastic Interpolants 2023) and (Song et al. Score-based generative modeling through SDE. ICLR 2021) matter to our own work here and becomes one of reasons our own work gets low rated. In our appraisals of these two works, both are excellent and insightful works by utilizing the continuous-time models, and each has its own perspectives toward the generative models: one used the time-reversed SDE and the other directly parametrized a path between two random variables. + “considering this work is only a small subset of [2]” Here [2] refers to (Albergo et al., Stochastic Interpolants 2023) which first appeared at arxiv on 15 Mar 2023 (two months before our submission to NeurIPS). Our original submission has mentioned this reference on page 2. Even though these contemporaneous works are in the large field of diffusion and generative models, they clearly focused on different problems and adopted entirely different techniques. We strongly disagree that our work is only a small subset of [2]. + **The problems solved in our work and [2] are different:** Stochastic Interpolants (2023) derives the continuity equations governing a stochastic interpolant by identifying the underlying velocity field. Our problem here is that within the standard score-based diffusion model, how the KL accuracy is impacted by the score error, under different diffusion coefficients. + **The techniques are different:** Theorem 2.21 in (Stochastic Interpolants 2023) is to directly estimate the KL error by two loss functions for their velocity field and score function. Our approach works on the first variation (functional derivative) of the KL error w.r.t. the perturbation of score. The detailed mathematical techniques and skills behind these works are also completely different. + **Our main result is not covered by (Stochastic Interpolants 2023):** In stochastic interpolants (2023), even though Section 2.4 shows the upper bound of sampling error in the setting of stochastic interpolant minimizing two vector fields and Section 3.2 has some discussion on deterministic vs stochastic generative models, there is indeed no similar conclusion for the score-based diffusion model as our main results here. Our results indicate that it is not only the loss of the score function matters, but *the distribution (in time) of the score errors matters for the choice of the diffusion coefficient*.
1. Can reviewer LAed please elaborate on why this work is only a small subset of [2], in light of the differences listed in the authors' responses? It would be most helpful if you can provide some examples of "high similarities in both techniques and results". 2. As noted by the authors, [2] is posted to arxiv two months before the NeurIPS deadline. I think it is fair to consider these contemporary works. Does reviewer LAed's opinion on the merits of this paper change *if we discount potential similarities to [2]*? Thank you for your input!
response
Thanks for the detailed response and clarifications!
Dear reviewer, The author-reviewer discussion period ends in 2 days. Please review the authors' rebuttal and engage with them if you have additional questions or feedback. Your input during the discussion period is valued and helps improve the paper. Thanks, Area Chair
Dear reviewer, The author-reviewer discussion period ends in 2 days. Please review the authors' rebuttal and engage with them if you have additional questions or feedback. Your input during the discussion period is valued and helps improve the paper. Thanks, Area Chair
Dear reviewer, The author-reviewer discussion period ends in 2 days. Please review the authors' rebuttal and engage with them if you have additional questions or feedback. Your input during the discussion period is valued and helps improve the paper. Thanks, Area Chair
I understand that the nature of the paper requires a large amount of equations and variables and I agree that adding more explanations/motivations will enhance the readability of the paper. I appreciate the authors revisiting this. I thank the authors for pointing out areas in which they addressed the limitations and potential societal impacts of their work. With these, I am comfortable moving my rating to a 5.
Decision
Accept (poster)