Star-Shaped Denoising Diffusion Probabilistic Models

Denoising Diffusion Probabilistic Models (DDPMs) provide the foundation for the recent breakthroughs in generative modeling. Their Markovian structure makes it difficult to define DDPMs with distributions other than Gaussian or discrete. In this paper, we introduce Star-Shaped DDPM (SS-DDPM). Its star-shaped diffusion process allows us to bypass the need to define the transition probabilities or compute posteriors. We establish duality between star-shaped and specific Markovian diffusions for the exponential family of distributions and derive efficient algorithms for training and sampling from SS-DDPMs. In the case of Gaussian distributions, SS-DDPM is equivalent to DDPM. However, SS-DDPMs provide a simple recipe for designing diffusion models with distributions such as Beta, von Mises$\unicode{x2013}$Fisher, Dirichlet, Wishart and others, which can be especially useful when data lies on a constrained manifold. We evaluate the model in different settings and find it competitive even on image data, where Beta SS-DDPM achieves results comparable to a Gaussian DDPM. Our implementation is available at https://github.com/andrey-okhotin/star-shaped .

Paper

Similar papers

Peer review

Reviewer 7FuV8/10 · confidence 4/52023-06-25

Summary

This introduces the so-called Star-Shaped DDPM (SS-DDPM). Instead of using Gaussian forward diffusion step in a Markovian manner, the diffusion process is directly conditioned on the data, in order to construct the "true" reverse process (posterior), resulting in the star-shaped diffusion process. This opens the door for using other transition process as demonstrated in the paper.

Strengths

1. This is a solid paper that I have reviewed so far. All the conclusions are backed by the theory. 2. The paper is well presented and easy to follow the main the idea with full materials in the supplementary 3. The idea presented in the paper is novel and may have wide applications.

Weaknesses

Frankly speaking, I like to see this paper to be accepted. I cannot form any weakness. The only point I may raise is it would be much much better for the authors to share their implementation source code.

Questions

When using other transition distributions in the forward process, what is the strategy to actually diffuse the data (into noises)?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Not much in this perspective. No comments

Reviewer P24X6/10 · confidence 4/52023-07-03

Summary

The paper introduces a new probabilistic structure for denoising models that does away with a Markov forward process. The authors derive the reverse process in terms of a sufficient statistic that allows for efficient reverse sampling. The form of the model is derived for a variety of noising distributions and the model is evaluated on a range of different data modalities.

Strengths

This work can potentially help many people in the generative modelling community because there has been a lot of work trying to extend corruption process to different data types e.g. discrete, manifold data and this paper has a general recipe that appears to work for a large chunk of these and more which could be really helpful when encountering non-standard data types. The example of the Wishart distribution on positive definite matrices is very cool. I can foresee many extensions to this work, looking to bring more distributions into the framework, extending it to continuous time which could bring up a lot of interesting technical problems and broadening the class of possible distributions by introducing approximate sufficient statistics. The paper has a good suite of experiments on a variety of real world practically relevant data such as text and images and the performance of the model seems good

Weaknesses

I think the paper could benefit from more intuitions and visualizations for the role of G. Initially, the model seems strange as x_t can jump a lot between x_t and x_{t+1} due to going to x_0 and resampling q(x_t | x_0). However, it seems that G instead is the smoothly changing statistic that is more analogous to the state in previous diffusion models. For example, I think figure 9 in the supplementary is quite interesting and could be moved to the main as it shows that a correctly normalized G looks very similar to the state in norm diffusion models. It would be good to have more intuition as to the role of G and how it is more closely linked to the 'x_t' of normal diffusion models. Further, there is not really a experiment that really motivates the use of these more interesting distributions in a scenario where normal diffusion models really don't work at all. I appreciate that this is somewhat an open question at the moment in the literature more generally and the community has already proven an interest in adapting the corruption process to the data modality of interest so this may not be a big requirement. Edit after rebuttal: I have read the authors rebuttal and appreciate the answers to my questions about the derivations and the FID scores. I intend to keep my score as it is.

Questions

In the proof of Theorem 1, how do you go from line (23) to line (24), as in, how do you know what q(G_t | x_0) is, for the numerator in (24)? Figure 6 is confusing to me because the figure makes it seem like the model is better than DDPM at all NFE but in the text the best FID is stated to be 3.17 which is the same as the best FID from DDPM and so shouldn't Figure 6 show the method and DDPM at least overlapping at 10^3 sampling steps? In DDPM, the Gaussian form of the reverse distribution is justified by considering the limit as the noise in each step is taken to 0 and the number of steps is taken to infinity. Is there any such justification for the Gaussian form that you assume in your model or is this a heuristic?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations of the framework e.g. restriction of the distributions and ability to find the sufficient statistics was nicely discussed.

Reviewer EaR35/10 · confidence 5/52023-07-05

Summary

This paper proposes a star-shaped diffusion probabilistic model, which is non-Markovian, and more like an autoregressive way of predicting intermediate states xt. Specifically, the authors define a forward process q(xt|x0) that each intermediate state xt is directly related to the initial state x0. The reverse process is defined by p(xt|xt, xt+1, ..., xT). To efficiently solve p(xt|xt, xt+1, ..., xT), the authors propose a compressive representation Gt(xt, xt+1, ..., xT) to extract all the information of x0 contained in history states {xt+1, ..., xT}, so the modified distribution becomes p(xt|Gt). The training and inference go in a similar way as DDPM. Simple experiments show that the proposed method performs better than the baseline methods MTD and DDPM.

Strengths

1. This paper provides a novel view of the definition of diffusion models and is very inspirational. It is worth thinking about whether the Markov process is the best design choice considering the fact that the whole history {xt, xt+1, ..., xT} contains more information than a single state xt. 2. The author proposed a novel diffusion model with basic evaluations, which may inspire the community for further exploration.

Weaknesses

1. The presentation needs to be improved: (1) T, Ωt, and ht in Eq.13 lack interpretation; (2) It is not well proved that Gt covers all the information of x0 in {xt, xt+1, ..., xT}. Although the authors mention that there are explanations in the supplementary material, you need to ensure the reader has a basic understanding by just reading the paper. 2. The advantage and application value of the proposed method is not clear. For example, can you provide a practical case that DDPM can not address while SS-DDPM can? 3. The experiments are too simple and not persuasive. To prove the advantage of SS-DDPM, it is encouraged to compare with state-of-the-art methods on diverse benchmarks, 4. As the authors said in line 128, "In general case the dimensionality of Gt would grow with the size of the tail" 4. There is no evidence that SS-DDPM can apply acceleration methods like DDIM.

Questions

I doubt the reverse process of q(xt|x0) should be p(x0|xt), can you explain why the reverse process is p(xt|xt, xt+1, ..., xT)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

The scope of this paper is interesting, but it needs more time to polish the theory, presentation, and experiments. Considering its current form, I think this paper does not meet the acceptance criteria of NeurIPS. However, if there is enough evidence to show the value of this paper, I may also change my score.

Reviewer mgR27/10 · confidence 4/52023-07-06

Summary

This paper proposes a non-Markovian diffusion model named the star-shaped diffusion model that generates a sequence of noised image from the original image in the forward process. This paper studies the theoretical foundation of such new type of model showing that if the forward process is based on a subset of the exponential family of distribution, a tail statistics is sufficient statistics for the reversed process. Based on such a foundation, the reversed process can be conducted.

Strengths

The overall structure of this new diffusion model is new, novel and interesting. The application of PKD theory is sound.

Weaknesses

Although the model is pretty novel and new, there is a lack of motivation from practical problems -- the experiment are weak and kind of toy. A showcase of the unique application of this model is appreciated. Lack of comparison of some other non-markovian diffusion on constrained domains such as [1,2] [1] Learning Diffusion Bridges on Constrained Domains [2] First Hitting Diffusion Models for Generating Manifold, Graph and Categorical Data

Questions

Is there any ODE-version of this SDE-based model?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See above

Reviewer UBua8/10 · confidence 4/52023-07-09

Summary

The paper presents the Star-Shaped DDPM (SS-DDPM), a general recipe for designing a diffusion model with a noising process lying in a general subset of the exponential family. With a Gaussian noising process, SS-DDPM recovers the DDPM. Diverse experiments on synthetic and practical image and text datasets demonstrate the effectiveness of the proposed SS-DDPM.

Strengths

(1) A general recipe for designing a diffusion model, termed Star-Shaped DDPM (SS-DDPM), is proposed. (2) The recipe is derived and analyzed in detail, with convincing statistical justifications. (3) Diverse experiments are conducted to demonstrate the effectiveness of the SS-DDPM.

Weaknesses

(1) Many technical details are given in the supplementary material. (2) The advantages and disadvantages of the SS-DDPM over the DDPM are not explicitly discussed.

Questions

What are the limitations of the proposed SS-DDPM, when compared with other diffusion models?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reviewer EZ376/10 · confidence 4/52023-07-17

Summary

This paper proposed a star-shaped denoising diffusion probabilistic model (DDPM), which extends DDPM to non-Gaussian noises. As a result, the backward/generative process requires conditioning on tails. The authors then propose an efficient tail conditioning strategy which works when the forward process follows an exponential family with linear parameterization. Duality between star-shaped and Markov diffusion processes are also established which provides theoretical support for star-shaped DDPM for its ability to go beyond Gaussian noises. The effectiveness of star-shaped DDPM is demonstrated on several experiments.

Strengths

1. This paper is written very clearly and well organized. 2. It also provides a general framework for DDPM that can incoporate no-Gaussian noises, where previous efforts are designed for specific noises and not generalizable to other distributions. 3. Duality between star-shaped and Markovian diffusion is established, which shows SS-DDPM and vanilla DDPM are equivalent in the Gaussian case.

Weaknesses

1. While experiments on synthetic data demonstrate the ability of SS-DDPM to deal with non-Gaussian noises, the effectiveness of SS-DDPM on real data needs further verification. The current experiment on real data seems a bit inadequate. 2. Star-shaped denoising diffusion models have been proposed before (Rissanen et al. (2022)), and there is no comparison to this previous method.

Questions

1. How does DDPM perform on the Geodesic data in the experiments? 2. How much improvement can SS-DDPM have over DDPM on real data? The advantage over DDPM seems a bit tiny based on current evaluation. Also, in which case would non-Gaussian noise be useful for DDPMs? 3. Rissanen et al. (2022) also proposed a similar star-shaped DDPM, where the reverse process is Markovian based on the heat equation. How does your method compare to their's?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, they do.

Reviewer P24X2023-08-11

Thank you for the response. I understand now how to get to line (24), I think just the trick of renormalizing wrt x_0 is what I was missing. For the FID results, my worry was that the DDPM results might have been calculated with 10K samples whereas others with 50K and when you use less samples in FID it biases the result upwards. But it seems in Nichol&Dhariwal 2021, they were using 50K for this plot so I think the results still stand. I will keep my score as it is currently.

Reviewer EZ372023-08-16

Thanks for the response, and I will keep the score.

Reviewer UBua2023-08-20

Thanks for the detailed responses. I will increase my rating from 7 to 8.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC