Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

Summary

The paper presents a general method for learning to de-corrupt datapoints in order to perform generative modelling. The paper proposes a sampling method that applies iterative updates to the current state that significantly outperforms a naive sampling algorithm which reconstructs and denoises alternatively. The authors present a wide variety of corruption processes and outperform a single step model and the naive algorithm.

Strengths

Extending the idea of creating a generative model from corrupting data and learning to reconstruct it to a more general framework is a problem I believe many in this field are interested in. Not least for me personally, I have been interested in this problem and their sampling algorithm would have been useful for me to know when investigating this topic. This paper takes a very general approach making very few assumptions on the corruption process and makes a non-trivial contribution by coming up with a sampling algorithm that can perform iterative updates to generate new data which is found to be crucial for this type of method to work well. The paper and methodology have flaws as I discuss next but I do believe that researchers will build on this work and find this contribution useful. The line of work is more engineering focussed and less based on theory which is not necessarily wrong and I think the novelty and likely interest in the topic slightly outweighs the negatives in this case.

Weaknesses

Starting out, the performance of the method is just not that good in terms of sample quality. I appreciate that getting methods to work well takes development over the course of multiple papers but I worry that this is a limitation of the method itself due to most corruptions tested not working well compared to standard diffusion models. When using this framework as a generative model there appear to be issues when the deterministic corruption transformation is not 1-1 due to the corrupted space being much smaller e.g. the space of images with constant colour or completely blurred images. The authors report problems with reduced diversity because of this and need to add a little bit of noise to increase diversity. I think this is quite a major flaw in the design because unlike standard diffusion models that can rely on maximum likelihood arguments to enable coverage of the data distribution, there are no such guarantees here and it is not clear how much the noise trick alleviates the issue. This flaw seems quite fundamental to some corruption process and a proper investigation into the diversity of samples would be good. Its a little strange how in effect all of the data distribution is being squeezed into this small subspace and then perturbing around the subspace a little bit induces diversity, this seems to be quite ill-conditioned in how small changes in input make large changes in generated output. The claims about new questions being raised as to the necessity of noise in generation should be reigned in because of the popularity and performance of flow matching type methods that are deterministic during training (given a randomly sampled pair) and sampling. The links to these methods should also be discussed especially in the case of corruptions such as animorphosis which is quite similar to a flow between two arbitrary data distributions and building a interpolating bridge between two random samples from them. The sections on deblurring and inpainting should be better introduced as the main paper talks about generative models and so this section comes as a bit of a surprise when I would have expected pure unconditional generation as the initial experiment. Typos: equation after line 145, need .e on final line proof A.9, need i = t-1 on third line of E_t^2. How do you move the sum out of the norm on the final line ? Edit after rebuttal: I have read the author's rebuttal and as I mention in my reply, my points regarding the ill-conditioned nature of the generative process and poor performance are still concerning for me and I intend to leave my score as it is.

Questions

The inpainting results seem so much better than other results in terms of sample quality. When you say Figure 4/Figure 10 is showing 'test images' do you actually mean that these were held out during training or has the network seen those during training. This should be made very clear since it seems that the network has just memorized those images (at least on celebA).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Discussed in weaknesses section.

Summary

This paper extends the Gaussian diffusion model toward arbitrary image-to-image translations, named Cold Diffusion. Specifically, the authors define a generalized forward diffusion process and its training process, then propose a novel Transformation Agnostic Cold Sampling (TACoS) process for generations. Experiments show that Cold Diffusion can effectively achieve image generation by learning image-to-image translation.

Strengths

This paper proposed a novel idea that the diffusion process can be applied to arbitrary image-to-image translations, not limited to Gaussian noise. The ideas proposed in this paper have achieved a certain impact in the field of diffusion models and inspired a lot of work. In light of this, I recommend that the paper be accepted.

Weaknesses

The author only provides an empirical formulation, without rigorous theoretical analysis. Nonetheless, this cannot overshadow the novelty of this work.

Questions

Is it possible to apply the diffusion process to any domain translation, i.e., dimension agnostic and modality agnostic?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

1. It seems that the image-to-image translation needs to have invariant dimensions. 2. It will be interesting to have a mathematical analysis of Cold Diffusion.

Summary

This work introduces a novel approach called cold diffusion, in which both the forward and backward processes are deterministic. The authors propose a scheme called Tacos, which predicts x_{s-1} from x_s by leveraging the estimated increment D(\hat{x}_0, s) - D(\hat{x}_0, s-1).

Strengths

The autors propsed nontrivial generalization of diffusion generative model to simple and straightforward deterministic process.

Weaknesses

1. The justification of TACos (Section 3.3) appears weak. - Higher-order terms may have a significant impact that is not adequately addressed. - TACoS is likely to fail in standard Gaussian diffusion scenarios. 2. As mentioned in Section 5.2, the generated samples exhibit low diversity. This indicates a failure to accurately recover the sample distribution, which is a primary objective of diffusion generative models. 3. Beyond the issue of diversity, the quality of the generated samples is also questionable. In the appendix, the 128x128 generated samples demonstrate significantly poorer quality compared to regular generative models.

Questions

The output quality of other cold generations is questionable. For instance, in the case of super-resolution, starting from a 2x2 image for the backward process may lead to similar issues of low diversity in the generated samples.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

This reviewer appreciates the authors' introduction of a novel deterministic generative diffusion framework. However, it is important for the proposed framework to demonstrate comparability to reasonable GAN models in terms of diversity and output quality. The inclusion of Gaussian noise (as seen in predictor-corrector models or Langevin dynamics) or a similar randomization process is typically considered necessary and serves as a key component in diffusion processes. To justify the diffusion without noise, it is crucial for the authors to provide results that are at least comparable to decent GAN models.

Summary

This paper introduces a method for image generation based on generic degradation and reconstruction operators. The approach generalizes diffusion models, which correspond to degradation by additive Gaussian noise, and reconstruction by denoising. In TACoS, the sampling scheme is agnostic to the choice of image degradation, and the corresponding restoration operator is learned via least squares regression over the data. The authors additionally introduce a sampling iteration with a correction term (Algorithm 2) that induces first order cancellation of errors induced by improper learning of the reconstruction operator. The correction term is shown to greatly improve performance over a naive approach, since it prevents blow-up of fitting error over multiple iterations. The authors also prove that in a toy problem (degradation via frequency filtering) that Algorithm 2 has smaller reconstruction error than the naive approach. Finally, the authors demonstrate that TACoS can be used for sampling and reconstruction with a variety of degradation operators, such as deblurring, inpainting, and superresolution. First, they show that in these cases, solving the reconstruction problem associated with each degradation is feasible for large-scale image datasets such as CIFAR-10 and CelebA. Then, they show that the blur transformation can be used to sample CelebA and AFHQ images, albeit with significantly reduced sample quality and diversity. They finally show as a proof of concept that other transformations such as inpainting, super-resolution, and animorphosis, can also be used to generate samples.

Strengths

- Clarity: the paper is well written and very clear. To the best of my knowledge the derivations are correct. - Novelty: the idea behind this paper is interesting and novel to the best of my knowledge. However, as I will discuss in below, the practical value of this idea is unclear. - Low computational cost: the proposed method is simple and computationally chip, given that it only requires fitting one regression model over the dataset. - Experimental methodology: the experiments in this paper clearly demonstrate that TACoS can be used for sample generation under a variety of datasets and image transforms. It is interesting that the method can be used to generate samples with arbitrary degradation and reconstruction operators, as opposed to Gaussian noising and denoising via score-matching.

Weaknesses

- Unclear practical value: in my opinion, the practical value of this paper is unclear because the proposed algorithm appears to have low sample quality and diversity. There are many existing methods for deterministic iterative sampling algorithms, notably flow based methods like Continuous Normalizing Flows [1] and Probability Flow ODE [2], which can both eliminate the need for sampling noise and attain high sample quality. - Unclear applications: it is interesting that the proposed method can use arbitrary image transformations, which may pave the way for other applications beyond sample generation. However, it is currently unclear what these applications may be, which further limits the value of this approach. - Lack of baselines: the sample generation experiments in Table 4 should also include baseline values, representing the FID that can be attained by existing methods such as vanilla DDPM or GAN based methods. [1] Building Normalizing Flows with Stochastic Interpolants (Albergo and Vanden-Eijnden, 2023) [2] Score-Based Generative Modeling through Stochastic Differential Equations (Song et al., 2021)

Questions

What are some potential applications (beyond sampling) for TACoS with non-standard degradation operators like animorphosis? Note: it's a bit confusing to report RMSE in Tables 1-3 but to then discuss the PSNR in the text. It would be helpful stick to one throughout the paper.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations and potential negative societal impacts of their work.

Response to rebuttal

Thank you answering my questions and for clarifying the equivalence between hot diffusion and DDIM. I appreciate the authors' efforts to discuss the merits of this paper with me. I feel that my understanding of the contributions and applications has not changed, so I do not plan to edit my score.

Clarification on my previous comment regarding Gaussian case

I continue to harbor doubts regarding its applicability to the Gaussian scenario, especially in the Variance Exploding setup defined by $x_t = x + W_t$ where $W_t$ is a standard Brownian motion characterized by $W_t\sim \mathcal{N}(0, t)$. For this case, Algorithm 2 fundamentally operates by subtracting noise: $x_{s-1} = x_s - Z$ where $Z = N_s-N_{s-1}$ is essentially a Gaussian noise. Given this setup, I'm skeptical about the algorithm's ability to produce accurate samples.