Understanding Hallucinations in Diffusion Models through Mode Interpolation

Summary

The paper introduce the concept of hallucinations for image diffusion models. The key contributions includes: 1. Definition of Hallucinations: Hallucinations are defined as samples generated by diffusion models that lie completely outside the support of the real data distribution. 2. Mode Interpolation Phenomenon: The phenomenon where diffusion models interpolate between nearby data modes, generating samples that do not exist in the original training data. This smooth interpolation leads to artifacts, termed hallucinations​​. 3. Causes of Hallucinations: Hallucinations are attributed to the smooth approximation of discontinuous loss landscapes by the diffusion model. This leads to interpolation between distinct data modes that are not present in the original dataset​​. 4. Experimental Findings: Experiments with 1D and 2D Gaussians show that hallucinations occur due to mode interpolation, particularly between nearby modes. The variance in the trajectory of the generated sample increases towards the end of the backward sampling process, indicating out-of-support samples​​. 5. Mitigation of Hallucinations: A simple metric to capture the high variance in the sample trajectory can effectively remove over 95% of hallucinations during generation while retaining 96% of in-support samples​​. 6. Impact on Recursive Training: The removal of hallucinations has implications for the collapse and stabilization of recursive training on synthetic data. Experiments on datasets like MNIST and 2D Gaussians demonstrate the effects​​.

Strengths

1. The paper introduces Hallucinations in diffusion models, which is an under-explored area. The paper's focus on mode interpolation as a source of these hallucinations brings a fresh perspective. 2. The paper exhibits high-quality research through its comprehensive experimental design and thorough analysis. 3. The paper is well-written and clearly structured, making it accessible to both experts and those new to the field. 4. The significance of the paper lies in its potential impact on the development and refinement of diffusion models. By identifying and addressing the issue of hallucinations, the research provides valuable insights that could lead to more reliable and accurate generative models.

Weaknesses

1. SMLD [1] claims that `the scarcity of data in low density regions can cause difficulties for both score estimation with score matching and MCMC sampling with Langevin dynamics.` SMLD [1] addresses this problem by introducing slow mixing of Langevin dynamics. The datasets used in the paper have the same density for different modes. I think the hallucination phenomenon with different density for different modes should also been explored. 2. While the paper provides a robust analysis using 1D and 2D Gaussian datasets, its experimental scope is somewhat limited. These simplified datasets may not fully capture the complexity of real-world data distributions. Is the hallucination of diffusion models on real image distribution (such as face) helpful to its diversity due to mode interpolation? 3. While the paper touches on the implications of hallucinations for recursive training stability, this discussion is relatively brief and lacks depth. Given the potential significance of this aspect, a more extensive exploration of how hallucination mitigation affects recursive training dynamics would have been valuable. This could include detailed experiments and analyses on the long-term effects of hallucination removal on model performance and stability. [1] Generative Modeling by Estimating Gradients of the Data Distribution

Questions

Please see weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have adequately described the limitations and potential negative societal impact of their work.

Summary

This paper addresses the *hallucination* phenomenon in diffusion models, where generated samples fall outside the training distribution. The authors propose *mode interpolation* as an explanation. They analyze synthetic datasets and conclude that hallucinations occur between nearby modes due to the inability of deep networks to learn ground truth score functions with sharp jumps for small timesteps $t$. The authors introduce a metric along the denoising trajectory to identify hallucinated samples and demonstrate that filtering out these samples during recursive training improves the quality of generated samples on both synthetic and real-world datasets.

Strengths

1. The paper explores a novel phenomenon of hallucinations in diffusion models and provides a plausible explanation through mode interpolation. 2. The authors employ sound toy experiments to validate the hallucination phenomenon and their proposed explanation.

Weaknesses

1. **Mode interpolation as an explanation for hallucinations is not entirely convincing.** The concept of mode interpolation is proposed and validated through experiments on synthetic Gaussian mixture datasets. However, the relationship between mode interpolation and hallucinations in more complex datasets and models (such as extra or missing fingers in StableDiffusion) remains unclear, particularly for latent diffusions with decoders. The authors should provide more evidence of mode interpolation in complex cases and hallucinated samples. Furthermore, the role of the decoder in hallucinations, mentioned in the abstract, is not adequately addressed in the discussion. 2. **Experiments should be more comprehensive and robust.** - *Comprehensive experiments:* The authors primarily use synthetic Gaussian mixture and simple shapes datasets to validate mode interpolation and hallucination removal. Additional experiments on more complex datasets like CIFAR-10 or CelebA, which are standard benchmarks for diffusion models [1], are necessary to generalize the proposed metric. Examples from more complex models, such as StableDiffusion, should also be included. - *Robust experiments:* The authors select dataset-dependent ranges of timesteps to calculate variance based on prior knowledge, which may not be robust. They should provide principles or guidelines for selecting the range of timesteps for different datasets. 3. **Insufficient analysis in some areas.** - *Sub-approximation of score function leading to mode interpolation:* The claim that mode interpolation is caused by the inability of deep networks to learn ground truth score functions with sharp jumps for small $t$ needs support. A "sanity check" experiment using ground truth score functions is necessary but missing, making the claim less convincing. The authors should include this experiment and analyze the approximated score functions in cases with varying data sizes, as in Figure 2. - *Relationship between hallucinations and high variance along the trajectory:* The authors argue that hallucinated samples exhibit high variance along the denoising trajectory, as observed in Figure 5. This appears more empirical than intuitive. More discussion is needed to explain why hallucinated samples have high variance along the trajectory. 4. **Lack of formal definition for Eq. (4).** The definition of $\texttt{Hal}(x)$ in Eq. (4) is informal (and inconsistent or incorrect): (a) The summation of $i$ over $[0,T]$ contradicts the range used in experiments; (b) The calculation of $\overline{\hat{x}_0^{(t)}}$ is unclear, and the superscript $^{(t)}$ may be incorrect; (c) There is a typo where $i$ should be replaced by $t$. The authors should provide a formal definition for $\texttt{Hal}(x)$ to make the metric more rigorous. [1] Denoising Diffusion Probabilistic Models, NeurIPS 2020.

Questions

1. **How do studies on properties of mode interpolation relate to its cause or detection?** In Section 4, the authors discuss properties of mode interpolation, such as its occurrence between nearby modes and the effects of data size and denoising timesteps. How do these properties relate to the cause or detection of mode interpolation? Could the authors provide more insights into the cause of mode interpolation and how these properties are related? 2. **How can a threshold for $\texttt{Hal}(x)$ be determined?** In Section 5, the authors propose $\texttt{Hal}(x)$ as a metric to detect hallucinations. What value of $\texttt{Hal}(x)$ is considered a hallucination in the experiments? Could the authors provide more insights into determining the threshold for $\texttt{Hal}(x)$? 3. **Does the sampling algorithm affect hallucinations?** The sampling algorithm details are missing from the experiment settings. Could the authors provide more information on the sampling algorithm used in the experiments? Additionally, do different sampling algorithms (e.g., deterministic vs. stochastic, as discussed in [1]) affect hallucinations? [1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.

Summary

This paper demonstrates and studies a particular failure mode of diffusion models termed mode interpolation. Specifically, the authors discovered that when trained on certain datasets, diffusion models (even those with 1000 denoising steps) generate samples that look like certain interpolations of some training samples. The paper demonstrates the mode interpolation effect on several toy datasets (e.g., 1D and 2D Gaussians, grids) and the MNIST dataset. The paper then delves into analyzing the cause of the mode interpolation behavior by examining the learned score functions. The authors observe that one cause of the artifact is that the denoising neural network cannot accurately mimic the score when it has abrupt changes. The paper finally proposes a metric to estimate the plausibility of a sample being a mode-interpolated one based on the observation that “good” samples often have relatively small changes in the later sampling process.

Strengths

The paper elaborates on a previously overlooked failure mode of diffusion models called mode interpolation. Specifically, when trained on certain datasets, diffusion models can generate samples that correspond to the interpolation of certain training samples. This phenomenon is not well-studied in prior work and this paper could bring further attention of the community to this problem. The paper is also very well-written and easy to follow.

Weaknesses

Despite the interesting observation of the mode interpolation behavior, my main concern is that the paper does not provide justifications for the mode interpolation behavior on more realistic datasets (e.g., natural images) and large models. While I totally understand that it is a major contribution to discovering the mode interpolation phenomenon, and it can be hard to observe this clearly on natural image datasets, I still believe some analysis can be done more carefully to let us understand mode interpolation better. I am happy to increase my score if the following concerns/questions are addressed. In Figure 1, while the Gaussian example clearly indicates interpolated out-of-distribution samples, the SIMPLE SHAPES results do not seem to support the “interpolation” behavior. Specifically, by interpolating training images, we will always get blurry or gray shapes, but in all sampled images, the individual objects seem to be perfect in terms of both shape and color. It seems like the diffusion model is doing some sort of ``compositional generalization’’ over the training samples. While this is clearly problematic in this synthetic dataset, this could be a good behavior for diffusion models on natural images as they can generate unseen object combinations. This will not harm but actually improve the model. I think the case where diffusion models fail is when they generate “interpolation of training images”, in which case the image will be blurry or has other artifacts. How do the denoising network structure and the forward noising schedule affect mode interpolation? Looking closely at Figure 2, noting the log scale of the y-axis, it seems that a modest amount of samples (e.g., 50000) is sufficient to almost prevent mode interpolation from happening (1000 times less likely). I wonder will mode interpolation happen more often or less often when we increase the dimensionality to be similar to e.g. natural images. In Section 4.3, the paper discusses the cause of the mode interpolation problem: the learned score function is inaccurate when the ground truth score changes significantly. It is still unclear how the sampling process interplays with this problem: does more steps mitigate or exaggerate the problem; and can other sampling methods (e.g., ODE-based) mitigate the mode interpolation problem? In Section 5.2, can we apply a similar metric that does not require T sampled trajectories? Since in practice we only draw a few samples a time (e.g., per prompt).

Questions

Please refer to the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed certain limitations in the appendix.

Summary

This paper studies the hallucination phenomenon in diffusion models, in which samples out of the support sets are generated. Specifically, the authors characterize a failure mode, termed mode interpolation, which is hypothesized to be attributed to the learned score function of the diffusion model being over-smoothed around the discontinuous jumps in the ground-truth score. The authors provide evidence for this hypothesis in experiments with 1D and 2D Gaussians, where the ground-truth scores are known. They then discovered the x0 predicted by the DDPM has higher variances (along the reverse diffusion trajectory) when generating hallucinated samples. Using this variance as a metric, the authors further show that they can filter out some hallucinated samples in datasets including Gaussians, SimpleShapes, and MNIST. They also demonstrate an application of such removal in recursive generative modeling, where samples generated by the current model are used to fine-tune the model.

Strengths

1. The hallucination problems of diffusion models are less explored than those with LLMs. This is an interesting research direction. It is interaction with another research direction, recursive generative modeling, as highlighted by the authors, is also non-trivial. 2. I like the authors' thought process in approaching this question. Regions between modes are indeed a reasonable starting point for investigating a likelihood-based model. 3. The proposed metric for detecting hallucination appears to be effective in the three synthetic datasets.

Weaknesses

1. It looks like only a particular parametrization of a particular type of diffusion model is tested, see Questions. 2. None of the datasets are natural images. Although the authors alluded to the analogy between mis-combining shapes and hands with 6 figures, the latter was not checked with real experiments. Would it be possible for the authors to apply the proposed hallucination metric to the hand generation problem and report some preliminary results?

Questions

1. Though the over-smoothed score function is shown in Fig. 4, it is unclear if this phenomenon is general. From what I can gather, the authors only work with DDPM, which by design is not a rigorous likelihood-based model. The weighting between noise levels is adjusted to promote perceptual quality, compromising likelihood estimate. I request the authors to try some more rigorous likelihood-based diffusion models, e.g. Variational Diffusion Models, to further verify the generality of their discovery. 2. Even with DDPM, there can be different types of model parametrization, such that the neural networks are learning with different targets. The default setting for model parametrization in the GitHub repo is epsilon prediction, which naturally has higher variance around low noise levels (i.e. timesteps close to 0). Did authors try x-prediction and v-prediction, two popular alternatives to eps-prediction? Specifically, variances in the supervision targets of the neural network should be low around low noise levels for x-prediction, and static for v-prediction. I believe these ablation studies can help further justify the generality of their discovery.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

I don't think the limitation of the proposed hallucination metric is sufficiently discussed. But I believe there should be negative societal impacts.