Summary
This paper proposes a method that leverages generative diffusion for image restoration tasks. The authors suggest incorporating the residual term, defined as the difference between the corrupted and clean images, into the forward and reverse diffusion processes. The forward process is described by a Markov chain, where the probability of the image at each time step $t$ is conditioned on the previous step, $t - 1$, and the residual term. This results in a forward where at each time step $t$, the image obtained at the previous time step $t - 1$ is mixed with the residual term and an additional white Gaussian noise. Consequently, the image at each time step can be described as a weighted sum of three components: the clean image, the residual term, and an additive white Gaussian noise.
Since the residual term is the difference between the corrupted and clean images, this weighted sum can be rewritten as a weighted sum of the clean image, corrupted image, and Gaussian noise. The forward process stops when the weight of the clean image becomes approximately zero. Thus, the forward process gradually transfers a clean image into a weighted sum of the corrupted image and white Gaussian noise. Correspondingly, the reverse process starts from this weighted sum and incrementally reduces the corruption and noise. Because the corrupted image is available, the initialization for the reverse process is easily obtained by mixing the corrupted image with white Gaussian noise.
The authors derive the expression for the mean of the reverse process and train a network to predict, at each step, the difference between the corrupted noisy image obtained at that step and the clean image. The proposed algorithm evaluated thorough experiments on three reconstruction tasks: shadow removal, low light enhancement, and deraining, comparing the performance with recent competing methods. In all experiments, the authors apply five diffusion steps. The results show that the proposed algorithm generally outperforms the competitors while consuming fewer computational resources (multiplication operations times the number of diffusion steps).
Strengths
I believe the idea of conditioning on the residual term is original and significant. It allows deriving a reverse process that starts from the corrupted image mixed with noise rather than from pure noise, reducing the number of diffusion steps required for image restoration.
Weaknesses
- Several notations are used without being defined. For example, the definitions of $\alpha_t$, $\beta_t$, $\bar{\alpha}_t$ are missing. Although the paper generally follows the notations defined in [1], defining these notations in the current paper would make it easier for readers to follow the explanations.
- Some mathematical derivations are not detailed enough. For example, the proof in Appendix A.1 is not sufficiently detailed. Another example is derivation of $\mu_{\theta}$ in Equation (13), which is missing. It is completely fine to rely on derivations in [1] or [2], but I believe including the relevant parts of the derivations in the appendix would greatly help readers follow the derivations.
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
[2] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.
Questions
To provide a more comprehensive background, it might be beneficial to cite [3] which introduced a general framework for leveraging generative diffusion for solving image restoration problems where the observed image is contaminated by linear degradation and additive white Gaussian noise.
[3] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in neural information processing Systems 34:21757-21769, 2021.