Summary
Solving inverse problems using pre-trained diffusion models is challenging because the noise conditional measurements are unknown. Existing methods aim to approximate this unknown quantity using first- or second-order Tweedie’s formula. In this paper, the authors propose a hierarchical optimization procedure to get around this issue. Instead of explicitly approximating this quantity, the authors construct a non-Gaussian Markov approximation of the true diffusion posterior. The main idea is an extension of Red-Diff to a noisy posterior. Extensive experiments on pixel and latent diffusion models show that the proposed method is competitive to prior works in random inpainting, box inpainting, super-resolution, and Gaussian deblur tasks.
Strengths
1. Extending Red-Diff to a noisy posterior is a valuable idea and provides practical benefits, particularly when critical timesteps are *carefully* chosen in the reverse process.
2. Experimental results demonstrate improvements over prior works in tasks such as random inpainting, box inpainting, super-resolution, and Gaussian deblurring.
Weaknesses
1. How do you identify which timesteps in the reverse process contain the critical relevant information? Can the authors provide more details on how they select the critical timesteps, or if there is any analysis showing the impact of different timestep choices?
2. Line 368-372: The main contribution of the paper is a straightforward extension of Red-Diff, except that in this paper the posterior is assumed to be well-defined for noisy latents (which is already a very strong assumption). Given access to such noisy posterior, even the existing inverse problem solvers like DPS, PSLD or Red-Diff will perform relatively better. The main issue is that the posterior is not well-defined for noisy latents. It is defined only for zeroth noise level, which justifies these prior works (e.g. Red-Diff). So, the main contribution of this paper needs to be clearly articulated in the context of these existing methods. The authors are encouraged to provide a more detailed comparison or analysis showing how their approach differs from and improves upon Red-Diff and other existing methods, particularly under the stronger assumption of noisy posterior.
3. The paper is poorly structured, spending excessive space on background information. The main contribution only appears in Section 4.2 on page 7, leaving insufficient room for experiments. As a result, experimental results are presented without adequate explanation, appearing as statements for completion rather than detailed analyses. The authors are encouraged to move parts of the background and related works from Section 2 and 3 to appendix to make room for more detailed explanation of their contributions and experimental results.
4. Typo in Line 1100, 1223, 1348: Missing reference labels make it difficult to follow which table or figure is referred to in this context.
5. The authors should clarify how their experimental settings differ from those in the original baselines [1,2,3]. It is essential to discuss the rationale for these changes and the potential impact on the results, especially because the baseline results here differ significantly from those reported in prior works [1,2,3].
6. What do the authors mean when they say in H.5 VIPaint captures multi-modal posterior? For a simple Gaussian mixture, the exchange of expectation in Tweedie’s formula has a bias that will prevent VIPaint from sampling the right posterior. The authors are requested to provide a more rigorous explanation or proof of how VIPaint captures multi-modal posteriors, particularly in light of the bias that comes with Tweedie's (first-order) formula.
7. The results in Fig 18-24 either look very similar or distorted, which do not verify the claim that VIPaint samples multi-modal posterior. Could the authors provide specific metrics or analyses that could demonstrate the multi-modality of the sampled posteriors, rather than relying solely on visual inspection of the figures.
8. VIPaint uses a significantly higher computational budget than the compared baselines, such as DPS or PSLD. For a fair evaluation, the proposed method should be compared with state-of-the-art LDM solvers [2,3] that utilize (nearly) comparable computational resources.
9. The theoretical discussion in Appendix B covers well-known material and lacks new insights. The main contributions of this paper are introduced only toward the end. The authors are encouraged to present their main contributions earlier in the paper to clearly distinguish their work from established literature.
### Reference:
[1] Nazemi, A., Sepanj, M. H., Pellegrino, N., Czarnecki, C., & Fieguth, P. (2024). Particle-Filtering-based Latent Diffusion for Inverse Problems. arXiv preprint arXiv:2408.13868.
[2] Chung, H., Ye, J. C., Milanfar, P., & Delbracio, M. Prompt-tuning Latent Diffusion Models for Inverse Problems. In Forty-first International Conference on Machine Learning.
[3] Rout, L., Chen, Y., Kumar, A., Caramanis, C., Shakkottai, S., & Chu, W. S. (2024). Beyond first-order tweedie: Solving inverse problems using latent diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9472-9481).