On Calibrating Diffusion Probabilistic Models

Summary

This paper investigates calibrating diffusion models. They notice that the expected data score equals zero. Since models not usually do not learn this correctly, they introduce a calibration term to subtract from a learned score. The new objective now includes the expected model score and performs better compared to the uncalibrated models.

Strengths

The paper is well written and the method is presented clearly. The theoretical results are convincing. The main appeal of the method is its simplicity. The solver part of the results is solid.

Weaknesses

Since the focus is on post-processing, the method is best suited for discrete time diffusion (like DDPM). Otherwise, it is unclear how the expected values are saved for continuous t. Line 221 mentions the selection of t depends on the sampling schedule of the solver, but the solver then cannot be adaptive. The alternative of learning the network as in eq. 14 would solve this but seems overly complicated. As can be seen from Fig 1 the lines are simple so fitting some simpler model would make more sense. The results are showing the number of solver evaluations is lower and the likelihood is better. Likelihood is evaluated in a weird way. The only thing that is shown is that the expected learned score is not zero for "uncalibrated" models which is somewhat tautological. In this case, some other relevant metric like FID would be better. There is no empirical demonstration that the diffusion models are uncalibrated, in a similar way that confidence-accuracy diagrams show this for classification. One could learn some simple synthetic dataset from a known distribution. It is not clear why this would be a relevant task, as presented. I assume guidance would further uncalibrate the models but it is used for better samples. Perhaps the image domain is not the best choice here. Eq. 13 is presented as a fact, whereas it seems this is to be demonstrated. The main issue is the limited motivation for improving generative models on likelihood and a bad choice of experimental setup. Something other than images could solve both of these issues.

Questions

- Why is FID going up with the number of samples in Table 3 in both columns? - Can you incorporate this loss somehow during training so that the final model is calibrated? - Does this hold for any data distribution? - Could using 20 and 50 diffusion steps for generation be too little?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are addressed.

Summary

The paper proposes a simple procedure to improve the calibration of pre-trained diffusion models. Diffusion models have achieved strong practical performance, but its score estimation is often seen as a black box. The paper sheds some light on the issue, by detecting the inherent lack of calibration of the current state-of-the-art models, and proposing a simple procedure that reliably drives up the lower bound for model likelihood. Empirical evaluations showcase the efficacy of the method.

Strengths

The paper discusses the problem of calibration in diffusion models, a question long overdue in this area of research: while diffusion models have achieved eye-catching generative results, little is known about whether the score function is a loyal depiction of the ground truth, or even a gradient field in itself. The paper gives theoretically sound justifications of by framing the evolution of the score function as a martingale, and proposes a simple yet empirically quite effective way that adjust pre-trained diffusion models closer to the underlying score function. The overall writing and presentation of the paper is sound.

Weaknesses

I think Section 3.4 is unsatisfactory, as I think the proper conclusion should be that the uncalibrated DPM training objective do not inherently minimize the mean of the expected predicted noises. My 2 more specific comments are listed below. - The logic between the explanations in 3.4 and Figure 2 seems confounding. I have understood the main points made in the paper after several reads, so I think the paper could adopt an easier to follow explanation. I believe the author's main point is (i) the optimal score attainable via diffusion modeling is capped by finite training data, so while the optimal score $\nabla \log q_t(x_t;\mathbb{D})$ has expectation zero with respect to the marginal distribution conditioned on data, but it does not likely have expectation zero when given the entirety of $q_t$; and (ii) state-of-the-art diffusion models do not inherently minimize $\lVert\mathbb{E}_{q_t} s_\theta^t \rVert$. I believe those two points are quite separate but lines 188-191 present it as one singular point, which seems as if the paper treats the diffusion modeling as the gold standard of learning the scores, and that even the true score carries a bias in a finite-data setting. - I admit that the finite data bias can present a problem, but it is not what the calibration trick is for. Frankly speaking, the calibration trick does not manufacture new training data just because it is more calibrated, but only pushes the score estimate _towards_ the optimal score estimate attainable in a finite data regime, i.e., $\nabla \log q_t(x_t;\mathbb{D})$. It is with this argument that I believe the calibration trick will attempt to modify the score estimate even when presented with the ground truth $\nabla \log q_t(x_t)$. Therefore, I believe the main point is not about dataset bias, but that common score matching objectives do not take into consideration the minimization of $\lVert\mathbb{E}_{q_t} s_\theta^t \rVert$.

Questions

None.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

I believe that the authors can make a few improvements in its presentation of empirical results. I understand that page limit prohibits the addition of more content, but I believe that some plots deserve to be in the main body of text. - Figure 1 showcases the potential lack of calibration for pre-trained DPMs, but does not show the expectation of predicted noise _after_ calibration. I believe that one can show the effectiveness of calibration by comparing the 2 images and discovering the calibrated model has a much lower expected value of the predicted noise. - The paper could benefit from also presenting some generated images as it shows any improvements in the most straightforward way. Currently there is a figure in the appendix about how calibration "reduce ambiguous generations", but there are too many examples to see a measurable difference. I believe a smaller plot highlighting the said ambiguous generations deserves a spot in the main text.

Summary

This paper introduces a general calibration technique for diffusion probabilistic models (DPMs). The authors derive a time-dependent calibration term that is independent of any particular input under different model parameterizations. This calibration term can be computed in advance and repeatedly used for sampling. The experimental results demonstrate the effectiveness of the proposed calibration method with a noticeable reduction of FIDs when taking 15-40 NFEs.

Strengths

The proposed calibration method in this paper is novel, theoretically sound, and practically useful in that it in principle can be applied to all kinds of DPMs to improve sampling with little computational overhead. The authors provide a performance analysis with varying numbers of samples for approximating the calibration term using the MC method. The FID results suggest that 20K of MC samples are sufficient to estimate the calibration term and reach the best quality. In this sense, the pre-sampling computation seems not to be too expensive. Overall, this approach appears to be general and easy to implement. Therefore, I believe this paper is significant and worth acceptance in NeurIPS.

Weaknesses

Although the calibration term is not $x_t$-dependent, it depends on time ($t$) and the model parameters ($\theta$). I am therefore concerned about the stability of the proposed approach. From Figure 1, we can basically confirm that the range of calibration terms drastically varies over different data sets. In this sense, if we apply the proposed approach to a new data set, there are at least three dynamic factors -- $t$, $\theta$, and data distribution $q_0(x_0)$ that could affect the performance of this calibration method. In regard to this, I suggest the author conduct more validation experiments to prove the stability and consistency of the proposed method over different time steps, models, and datasets.

Questions

What are the FID results of AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64? Following my concerns on stability, in the main paper, the authors state that they have conducted experiments on these datasets based on pre-trained models considering their top performance. However, I cannot find these numbers in the main paper and in the appendix. Why do you choose to only present CIFAR-10 and CelebA 64x64 in the main paper? I believe that an improvement on the ImageNet 64×64 is more convincing. I suggest the authors keep the FIDs of all these datasets in the main paper and compare them to the original DPMs.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The impact of this paper may be limited by the lack of comparison against other existing methods. I tend to categorize this paper into research that improves pre-trained DPMs generally at inference time. Along this direction of research, there are several relevant works, which I believe the author could easily compare to, i.e., (not trainable) PNDM [1], Analytic-DPM [2], and (trainable) BDDM [3]. [1] Liu, Luping, et al. "Pseudo Numerical Methods for Diffusion Models on Manifolds." [2] Bao, Fan, et al. "Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models." [3] Lam, Max WY, et al. "Bilateral Denoising Diffusion Models."

Summary

This paper presents a simple way to calibrate an arbitrary pretrained diffusion probabilistic model (DPM), which can reduce the score matching loss and increase the lower bounds of model likelihood. The authors observe that the stochastic reverse process of data scores is a martingale, from which concentration bounds and the optional stopping theorem for data scores can be derived. The proposed calibration method is easy to follow and can be performed only once, resulting in models that can be used repeatedly for sampling. The premise is that DPMs are by default uncalibrated (which they show experimentally), and that calibration can result in improved generation. This again is showed experimentally using the FID metric. —- Raised score from 4 to 6 post-rebuttal. On the whole it seems like the method can have some positive impacts - mostly in terms of model likelihood (with subsequent downstream benefits), and sometimes in terms of the quality of generated images.

Strengths

- clear and concise presentation of the proposed calibration method - empirical validation of the method on multiple datasets, - derivation of concentration bounds and the optional stopping theorem for data scores

Weaknesses

This biggest weakness is solely relying on model likelihood and the FID score for measuring generative performance. FID (Heusel et al., 2017) calculates the Wasserstein-2 (a.k.a Fréchet) distance between multivariate Gaussians fitted to the embedding space of the Inception-v3 network of generated and real image. A major drawback with FID is its high bias. The sample size to calculate FID has to be large enough (usually above 50K). Smaller sample sizes can lead to over-estimation of the actual FID. This can clearly bee seen in Table 3. It's not a surprise that the model likelihood improves (since this is directly linked to calibration), but the link between model likelihood and generation quality is not at all clear.

Questions

In the supplementary material, some example generated images are given with/without calibration. I zoomed right in, but I struggled to see any difference between pairs of images - is there anything that demonstrates that calibrating improves matters?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors have partially addressed the limitations in that they acknowledged that the model likelihood comes down but not necessarily FID, although they don't answer the question of whether it improves generation quality! Also there is a scaling issue with the method that is identified.

Summary

The paper makes an observation regarding the reverse process of diffusion probabilistic model, noting that the data score term is a martingale with respect to this process. A key contribution of the paper is a theorem to this effect and the associated proof. Leveraging this observation, the authors propose a calibration technique: one can calibrate a pretrained diffusion model at any time step by subtracting its expectation. Experimental results in the paper demonstrate that the calibrated score model achieves lower values of score matching objectives. Furthermore, the paper provides evidence that the calibrated score model yields higher evidence lower bounds. Lastly, the paper explains that similar conclusions hold true for the conditional score model.

Strengths

S1. The paper makes a novel and significant observation concerning the martingale nature of the data score term in probabilistic diffusion models with respect to the reverse process. S2. The paper is clearly written, well-organized, and provides a key theorem and proof that are presented in a readily-followed fashion. S3. The innovative technique for calibration is soundly based on theoretically-derived principles, is relatively simple to compute, and achieves an experimental performance improvement. The method is extended to the conditional setting. S4. The paper provides visualizations of the expected predicted noises and associated discussion that helps the reader understand the inductive bias. The additional experimental analysis is a welcome addition (sensitivity to the number of training samples, performance using generated samples, dynamic recording performance).

Weaknesses

W1. Although the proposed method achieves an improvement in terms of model likelihood, the experiments are less convincing that there is a meaningful practical improvement in terms of generated samples. There is an improvement in FID, but the sample images are presented in the appendix in such a way that it is almost impossible to discern a difference between the calibrated and uncalibrated models. W2. The reported experiments only assess sample quality using FID. Multiple papers have demonstrated how this metric can be misleading in some circumstances and have proposed alternatives, e.g., precision and recall metrics, that can be used in conjunction with FID to provide a clearer and more complete assessment of the capabilities of a generative model. W3. There is very little discussion of computational and memory requirements for the calibration. The paper could benefit from a much clearer discussion of the costs that are incurred in order to obtain the calibration benefit.

Questions

Q1. Can the authors provide a clearer commentary about the claimed subjective quality improvement. The generated images in Appendix C are extremely small. There is almost no discussion beyond “Our calibration could help to reduce ambiguous generations” – this is almost impossible for a reader to verify with the way the images are presented and it is not clear exactly what is meant. Q2. Could the authors comment on how the method performs for other performance metrics for generative models? Q3. What is the computational and memory cost (particulary in comparison to the original model)? i.e., does it increase the computational overhead by 30 percent? Is the memory cost an additional 20 percent?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The paper provides a very brief discussion of the limitations. This is definitely useful, and the authors acknowledge some of the key limitations. On the other hand, it would strengthen the paper if the supplementary material contained a more in-depth discussion. In particular, additional discussion concerning overhead and whether the proposed method translates to genuine practical benefits would be welcome. The section could discuss the challenge of evaluating generative models and suggest potential research avenues fhat might help establish whether the calibration improvement translates to more applied settings.

Post-rebuttal response

I thank the author for explaining points I laid out in the reviews: I maintain the same score assessment for the paper as I already recommend acceptance.

Response to the authors

Thanks the authors for a detailed response and adding new experimental results. I am convinced that the proposed calibration trick is a general method applicable to diffusion models. I will keep my rating, leaning to an acceptance.