Summary
The paper addresses the issue of making diffusion models faster (faster sampling) while still yielding high quality, diverse samples, from a large phenomenological space (scaling to large datasets). This is achieved by a novel semi-implicit denoising diffusion model (SIDDM). The idea extends the approach followed in DDGANs by reformulating the objective, and a novel decomposition into two components: (a) a pair of marginal distributions over the denoised data (at each step) with an implicit form to be aligned with the JSD via a learnt critic, and (b) a pair of conditional, forward diffusion components having explicit forms to be aligned using the KL divergence. This mixed objective allows better scaling than DDGANs. A novel regularizer is also introduced for the discriminator which allows a more granular distribution matching. A proof of concept validation is performed using a synthetic, Mixture of Gaussians dataset. Ablations demonstrate the benefit of both the innovations. SIDDM is benchmarked against the art on the CIFAR10, CelebA-HQ-256, and ImageNet datasets demonstrating performance competitive with DDGANs, better scaling to ImageNet, and much higher sampling speeds than classical DDPMs.
Strengths
**Relevance** The paper addresses an important problem – speeding up diffusion based generative image models while not compromising on the image quality, with good scalability to large, complex datasets. This should be of relevance to the community working in these areas but also of interest to a much wider audience.
**Originality** The proposed decomposition is interesting and novel, balancing both a non-parametric, population-level statistical alignment of distributions and a direct (simple, parametric) objective which decomposes into a sample level objective over Auxiliary Forward Diffusion (AFD). It stands to reason that this can have a significant impact on the training of the diffusion reversal DNN.
**Technical Quality**
- The technical approach mostly appears sound. It creates a new, direct and stronger learning signal which augments the critic based learning objective. This can lead to better learning outcomes over more complex distributions representing larger datasets like ImageNet. In addition, regularizing the discriminator via the UnetGAN formulation seems reasonable too.
- The mathematical formalism used in the paper, including the upper bound in Theorem 1, add to the strength of the contribution, making it principled, and beyond the heuristic choices made. I haven’t verified the derivations in the paper entirely but have followed the structure of the derivations and the proof and find them reasonable.
**Experimental Validation** There are several positives in the choices made for evaluation:
- The choice of using a synthetic dataset based on Mixture of Gaussians (MoGs) allows for validating the core idea.
- While the results on CIFAR10 and CelebA-HQ-256 are comparable with DDGANs, the FID scores on ImageNet demonstrate a performance comparable to SOTA diffusion models while DDGANs performance seems to demonstrate a failure to scale to ImageNet. This validates the core motivation of the paper.
- The ablation study, with and without the AFD term, validates the benefit of the decomposition of the training objective that forms the basis of SIDDMs; as well as the benefit due to the UNetGAN regularizer in (14)
**Significance** The approach has the potential for significant impact over an important area of research once the approach has been reproduced and ‘hardened’.
Weaknesses
**Experimental Validation**
(a) The MoG experiment (Section 5.1) demonstrates that SIDDM achieves good results with very small number of time steps, and has better stability than DDGANs. However,
- It is not discussed why it is unable to converge to the simple MoG data distribution, when the number of time steps increases (similar to DDGAN).
- Table 1 shows the FID score. These trends don’t tally well with the behavior shown in Figure 3. Perhaps using other metrics (JSD, LPIPS, etc.) may explain the results better?
- The implications of the above on the learning on real datasets, and the scalability thereof on larger, complex datasets like ImageNet is unclear. Such tasks may require more time steps (it is not clear whether this is the case) in which case, convergence to the data distribution becomes an issue.
(b) The results on CelebA-HQ-256 are typically used to show performance on high-resolution data. The images (depictions) of the generated samples, both in the paper and in the supplementary, are too small. Similarly, the ImageNet depictions are very small.
(c) While NFE (number of function evaluations) and the wall-clock time are shared, no details of the hardware are shared.
**Clarity** (l. 230) It is not quite clear why the material on training with real datasets like CIFAR10, Celeb-HQ etc. is in Section 5.1 and not Section 5.2. Kindly fix.
**Reproducibility** Implementation details are missing. The authors also don’t mention that the code will be released. Given this, I suspect it may be hard to reproduce results.
**Discussion of Limitations** No discussion of limitations in the paper.
Questions
Kindly address the following:
- Explain the behavior on the MoG dataset using a larger suite of metrics (see Borji, CVIU 2019; and its 2021 update for example).
- Explain the behavior (non-convergence to data distribution) when using a larger number of steps and the scalability implications.
- Did you investigate the tradeoffs between a smaller step size with larger number of steps, and smaller number of steps with a larger step size? How does one decide this? Does the theoretical formulation lend itself to such a guidance?
- Kindly share details of the hardware, implementation, and other details which will aid reproducibility.
- Will the code be shared?
- Discuss limitations.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Authors don’t address limitations of the paper.
I don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.