Fast Samplers for Inverse Problems in Iterative Refinement Models

Summary

This paper introduces a inverse problem approach to reconstruct low-resolution blurred or obfuscated images, from pre-trained generative models. The method leverages conjugate integrators to project the diffusion dynamics in an easier space, solve the inverse problem and then map back.

Strengths

- The integration of Conjugate integrators for inverse problems in the paper is rigorously defined and proved. - The proposed method drastically reduces the number of sampling steps compared to other inverse problem approaches. - The presented approach achieve better reconstructed images compared with state-of-the-art inverse problem methods.

Weaknesses

The paper is extremely dense with equations and mathematical formulations. I would suggest moving sections 2.3 and 3.4, as well as the numerical details of the experiments, to the appendix. This would allow space in the main paper for additional qualitative results and a broader discussion on the intuition behind the conjugate integrators for inverse problems. Including a figure illustrating the method would also be appreciated. While the mathematical formulation is the main essence of the paper, a visual representation or descriptive insight would be appreciated for the understanding.

Questions

Minor details: - Line 608, link the approximation to Eq. (5)

Rating

7

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

I don't have any concern.

Summary

Inverse problems like super-resolution and deblurring remain computationally inefficient with current diffusion/flow models. The paper introduces a plug-and-play framework with Conditional Conjugate Integrators that use pre-trained iterative refinement models to simplify sampling. The method generates high-quality samples in as few as 5 steps for tasks like 4× super-resolution on ImageNet, outperforming existing methods.

Strengths

1. The paper is well-written, and the literature review is thorough. 2. The method is well-supported by theoretical foundations. 3. The experiments on both diffusion and flow-matching methods across various tasks demonstrate the robustness of the proposed speed-up algorithm.

Weaknesses

1. The reason why the paper is restricted to the $\Pi$GDM paradigm is unclear.

Questions

1. Can the method be applied to nonlinear inverse problems? If not, please clarify the reason. 2. Though the Conditional Conjugate Integrators work well and can speed up the sampling process to 5 steps, a more detailed computational comparison is needed as each iteration becomes more complicated. Can you provide more detailed computational results? 3. Can you clarify the reason why you restrict yourself to the posterior approximation in $\Pi$GDM? Though the experimental results are promising, a direct comparison with DPS is preferred.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the paper mentioned its broader impact at the end, which looks good to me.

Summary

This work proposes a plug-and-play based method that leverages pretrained diffusion and flow models to solve inverse problems. The proposed method called Conditional Conjugate Integrators adapts previously proposed Conjugate Integrators framework for fast sampling of diffusion and flow models to solve linear inverse problems. The key idea is to project the diffusion (flow) dynamics into another latent space where sampling can be more efficient. Upon completion, the dynamics is projected back to the original pixel space. The paper provides the mathematical forms of the projection operator for conditional diffusion (flow) dynamics, then adapts it to get the projection operator for linear inverse problems. The paper also provides tractable forms of the projection matrix and its inverse. This derivation can be seen as a more general form of previously proposed method $\Pi$GDM ($Pi$GFM). The method can be extended to nonlinear and noisy settings. The paper provides promising results on datasets like LSUN Bedrooms, AFHQ Cats, FFHQ, ImageNet etc. on tasks like inpainting, superresolution, gaussian deblurring etc.

Strengths

1. Methodology: The proposed method seems efficient — it can solve linear inverse problems in 5 steps. In comparison, previous methods need 20-100 or even 1000 steps. 2. Experiments: The paper includes ablation studies on the choice of hyper-parameters and provides comparisons against former state-of-the-art methods like $\Pi$GDM, DPS, DDRM,$\Pi$GFM etc. The proposed method out-performs previous methods by a significant margin while using 5-10 steps and is on par with or better than previous method for >=20 steps. 3. Writing: Paper is well written. The core ideas and the methodology have been presented well, and derivations are easy to follow.

Weaknesses

The benefits of the proposed method for the settings of noisy linear inverse problem setting as well as non-linear inverse problems remain unclear. The paper does not include any quantitative results for these two problem settings. It only provides some limited qualitative results for super-resolution for the noisy setting with $\sigma_y$=0.05 and compression inverse problem and JPEG restoration problem for non-linear settings.

Questions

Questions: 1. Why does Eq. 117 hold true for VPSDE? I’m missing the simplification step here. 2. Why does Table 1 not include results of inpainting for diffusion models on ImageNet? Similarly, Table 4 in the appendix skips results for inpainting on FFHQ. Suggestions: 1. All the tables and figures must explicitly state all the relevant settings of the inverse problems. It is not immediately apparent that some of these quantitative results are only for noiseless linear inverse problems. 2. Consider including an algorithmic box that summarizes C-$\Pi$GFM and C-$\Pi$GDM. This would provide a concise overview of the method to the readers.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discusses relevant limitations.