Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Summary

Authors propose ProjDiff which reframes noisy inverse problems with diffusion models as a two-variable constrained optimization by introducing an auxiliary optimization variable. Authors derive a two-variable ELBO as a proxy for the log-prior and solve the optimization problem via projection gradient descent. Authors conduct comprehensive experiments on several image restoration tasks (super-resolution, inpainting and deblurring), source separation, and partial generations tasks.

Strengths

* Overall, paper is easy to follow and structured nicely. * I appreciate extensive numerical results that are presented in the paper. * Numbers for source separation look significantly better than the baselines (I have to note that I'm not an expert in that area)

Weaknesses

* Prior work section is not comprehensive. * The gains in performance compared to baseline methods (especially for noisy restoration tasks) is not convincing. Moreover, hyperparameter search for baseline models were not conducted and wrong baseline method was used (DDNM for measurements with noise). * See the questions below.

Questions

* I have several questions regarding DDNM numbers. First of all, DDNM [1] is designed for noise-free image reconstruction problems. I believe that is the reason why DDNM on Gaussian deblurring performs bad (as in Table 1: 7db PSNR, 0.03 SSIM, etc.). I would suggest the authors to switch to the DDNM+ variant as described in [1] for noisy inverse problems. If authors are already using DDNM+, could you clarify why the performance is bad on Gaussian deblurring? * line 37-39: "However, it’s worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models’ denoising capability." Could the authors clarify what it means to not fully leverage diffusion models here? * I would recommend the authors to include $\Pi$GDM [2] in their comparisons especially since it performs much better than DDRM. * Some other missing citations on solving inverse problems with diffusion models: CCDF [3], latent diffusion models: PSLD [4] * line 663-665: "Since [23] did not conduct experiments on CelebA, we use the parameters on FFHQ for the CelebA dataset as both FFHQ and CelebA are datasets of faces.". I don't think this is a good practice. In my experience, even though both CelebA and FFHQ are face datasets, DPS is not robust to the choice of step size. I believe some hyper-parameter search on a small set of images is necessary for fair comparison. * Do you think ProjDiff can be extended for using latent diffusion models as a prior? --- References: [1] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. "Zero-shot image restoration using denoising diffusion null-space model." arXiv preprint arXiv:2212.00490 (2022). [2] Song, Jiaming et al. “Pseudoinverse-Guided Diffusion Models for Inverse Problems.” International Conference on Learning Representations (2023). [3] Chung, Hyungjin, Byeongsu Sim, and Jong Chul Ye. "Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [4] Rout, Litu, et al. "Solving linear inverse problems provably via posterior sampling with latent diffusion models." Advances in Neural Information Processing Systems 36 (2024).

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are adequately addressed by the authors.

Summary

This paper proposed a new sampling strategy for solving noisy inverse problems using diffusion models. The proposed method is called ProjDiff, which is based on two-step minimization of log-posterior using gradient descent. The sampling procedure is derived by (1) Reparametrization of the noisy measurement as a noiseless measurement of an intermediate noisy sample in the diffusion process, and (2) obtaining a proxy objective from the variational lower bound.

Strengths

+ The paper's approach to addressing the noisy inverse problem using the reparamterization of measurement is interesting. + Deriving the sampling procedure by directly minimizing the evidence lower bound is innovative. + The performance is competitive with the SOTA methods in solving inverse problems.

Weaknesses

+ The paper is not well-written and it is hard to follow. Justification: 1. Theoretical analysis results lack organization. Example: Lemma 1 and Lemma 2 should be separated from the main proof. The proofs for the propositions should also be separated and distinctly expressed. 2. Theoretical results should be explicitly included. The proofs for Lemma 1 and Lemma 2 are not included in the text. If taken from a source, then the reference should be mentioned, otherwise, the proofs should be included. 3. Abstract includes some concerning statements. >Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks. Inverse problems could be solved using MMSE estimation. It is unclear what kind of optimization algorithm the authors refer to. > by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. The abstract does not reflect the method used. The statement is vague and could be inferred in many different ways. Revising the abstract to better reflect the proposed method can strengthen the paper. 4. The second and third paragraph of the introduction needs to be revised. Instead of briefly mentioning what each existing method does, the authors could state how their method is different from the literature and how it contributes to the literature. * The truncation of the gradient is based on an assumption and leads to an approximation in the solutions. Thus, the reduced computational cost should not be mentioned as a contribution of the paper, specifically when the authors have not included any experimental results regarding this. > 44: Through gradient truncation, we obtain an approximation of the stochastic gradient of the objective, effectively sidestepping significant computational overhead. Overall, the weaknesses of the paper are mostly related to the presentation of the paper and the blurred message, not the proposed method and effectiveness of it. Minor issues: > Line 19: Their remarkable ability to capture data priors enables effective guidance for solving inverse problems [6], which are widely exploited in image restoration. Guidance in inverse problems usually refers to enforcing data consistency with the measurement and relies on the forward model.

Questions

What does the author mean by weak observations? (Line 214) How does the algorithm look when used for nonlinear inverse problems? Can equation (8) be written for nonlinear measurements? How do the authors set the $\eta_1$, $\eta_2$, and equivalent noise level in the algorithms? Are the solutions sensitive to these parameters?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Partially addressed by the authors.

Summary

The authors present a new way of solving inverse problems using a diffusion model based prior. The key idea is that the forward process, which is to be undone, usually involves Gaussian noise. The authors utilize this by viewing the noisy observation as a as random variable at an intermediate time step t of the diffusion forward process. This allows them to formulate the inverse problem as a joint optimisation problem for x_0 and x_t given the noisy observation. As a by product, their method is able to solve inverse problems which do not involve noise as well. The authors evaluate their method and achieve superior results compared to various baselines.

Strengths

* I believe this is a principled new perspective on diffusion models for inverse problems and I am sure it can be potentially useful for many practical applications. * The method achieves remarkable results. * I very much appreciate the fact that the method can be applied to noise-free inverse problems, without having to artificially include non-existent Gaussian noise in the model.

Weaknesses

* The motivation, comparing the method to the more established MAP approach, could be a bit more clear. The authors write: "In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously." Why exactly is is describable to utilize the denoising capability of the model? Is the something wrong with the normal way of including the log of the forward model in the optimisation task? Why can we expect an improvement? Is it a problem of optimisation? Surely the way task itself is normally (e.g. [28]) formulated is correctly, or not? * Some aspects of the method were difficult for me to follow. This may be due to my inexperience with some of the involved mathematics. In particular, I wonder if the SVD based decoupling could be explained or illustrated differently to facilitate understanding.

Questions

Regarding the SVD composition, I believe I am missing an important aspect about how the noisy observation can be considered as a step in the diffusion process when it involves A. Let's say A corresponds to a convolution with a blur kernel. Would this not mean that we are considering a blurred noisy image to be part of the diffusion process. This would be incorrect, because the blurring would not be part of the training data statistics which would only include clean images, right?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

My understanding is that the method is limited to either image degradation processes that involve Gaissian noise, or are noise-free. Is this the case? If yes, this would be a significant limitation with respect to practical applicability, e.g. in settings with Poisson shot noise.

Summary

This paper proposes ProjDiff for solving inverse problems with pre-trained diffusion models. By deriving a two-variable ELBO as a proxy for the log-prior, this paper reframes the inverse problems as constrained optimization tasks and address them via the projection gradient method.

Strengths

1. The paper writing is clear and easy to follow. 2. The derivation of some formulas in this paper is solid.

Weaknesses

1. The assumption of this method for the measurement noise in Section 3.1 is doubtable. (1) As the authors claim in the limitation section, assuming the Gaussian noise will limit its applicability to other noise types like Poisson or multiplicative noise; (2) The whole method design heavily relies on know the exact standard deviation of the Gaussian noise \sigma, which is impractical and can lead to robustnesses issues when facing unknown noise level. Actually, the noise estimation itself is a challenging problem, particularly on the degraded measurement A(x). Please check the related works [1,2]. In general, this method is built on an impractical scenario, so I am worried about the practical usage of this method. 2. The approximation for implementing the update rules just after Proposition 2 is not convincing. (1) One of the reasons for this approximation is that "the μ-predictor of the diffusion model should be resilient to small perturbations in the input", but this hope usually is not the truth, particularly when t is large, i.e., at the beginning of the reverse sampling procedure; (2) There is not much evidence for this important approximation, no matter theoretical or empirical evidence. 3. The proposed method is too "delicate", as shown in Algorithm 1. (1) Again, this method needs to know the measurement noise level; (2) this method contains several hyper-parameters. In Table 17, it is clear that the hyper-parameters are heavily tuned, and there is no specific way to guide the hyper-parameter tuning. 4. No ablation studies for the hyper-parameters. For this "delicate" method, a systematical ablation study is needed. For example, it would be good to report the results for different combinations of some important hyper-parameters. 5. The extension to nonlinear inverse problems is not convincing. This paper proposes to use min_x || y - A(x) ||^2 to approximate the projection operator. However, it is almost impossible to solve nonlinear inverse problems by using this formulation because it will be easy to be stuck in local minimizers. Also, if this formulation can solve nonlinear inverse problems well, then there is no need to write this paper. 6. The experiment settings for phase retrieval are doubtable. (1) I highly suspect that this paper downplays DPS in some implicit ways. I ran DPS for phase retrieval on FFHQ before, and remembered DPS could achieve over 30dB for PSNR after trying different initializations. In the original DPS paper, the authors claim that DPS need different initializations but it seems that this paper omits this; (2) there is no comparison with the golden standard method for phase retrieval, i.e., HIO+ER. 7. Some recent SOTA methods are missing for comparison, including ReSample[3] (ICLR'24 spotlight), DiffPIR[4] (CVPR'23), DMPS[5]. Please check the recent survey for more related works [6]. [1] Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237. [2] Li, F., Fang, F., Li, Z. and Zeng, T., 2023. Single image noise level estimation by artificial noise. Signal Processing, 213, p.109215. [3] Song, B., Kwon, S.M., Zhang, Z., Hu, X., Qu, Q. and Shen, L., 2023. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123. [4] Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R. and Van Gool, L., 2023. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1219-1229). [5] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343. [6] Li, X., Ren, Y., Jin, X., Lan, C., Wang, X., Zeng, W., Wang, X. and Chen, Z., 2023. Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey. arXiv preprint arXiv:2308.09388.

Questions

See weaknesses.

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

See weaknesses.

Thank you for the rebuttal.

Thank you for the clarifications! **Regarding the motivation:** *"The conventional approach of including the logarithm of the forward model in the optimization task is also feasible, as seen in [24, 28] in the main paper, and we have compared with it in our experiments (RED-diff). The improvement of our algorithm compared to these methods can be expected because our method utilizes more information modeled by the diffusion model, i.e., it uses both the prior and denoising capabilities (or both the clean data prior and the noisy data prior). We believe this is the essential difference between our algorithm and the methods that include the log of the forward model and modeled by diffusion models in the MAP framework."* I am afraid I still can't completely follow here. Surely the correct prior together with the correct forward model should provide (or be proportional to) the correct posterior. Maximising the posterior will provide the true MAP estimate. How can *"additional information"* beyond this be useful? Are you saying your method is better at finding the MAP estimate (compared to the conventional approach) or is your method finding something better than the MAP estimate?

Response to the rebuttal

Thank you for providing the rebuttal. After reading the reviews from all the reviewers and rebuttals, I would update my score to 6.

Looking forward to your reply!

Dear Reviewer g7gD: We would like to express our sincere gratitude once again for your valuable comments and suggestions. The discussion phase is due to conclude in 30 hours, and we have already received responses from the other three reviewers. We are eagerly awaiting your feedback. Should there be any further questions, we would be more than happy to engage in continued discussion if time permits. Best regards.