Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs

Summary

This paper proposed a Domain Shift diffusion-based SR model, namely DoSSR, which capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. In addition, authors advance their method by transitioning the discrete shift process to a continuous formulation to further enhance sampling efficiency. The proposed method is evaluated on synthetic and real-world datasets and achieved competitive results.

Strengths

1. The proposed DoS-SDEs achieve a remarkable speedup in inference, demonstrating its superior efficiency. 2. The proposed method achieved competitive results on non-reference metrics. 3. The paper is well-organized and the figure is clear to understand.

Weaknesses

1. The Domain Shift scheme seems to be similar to ResShift, where noise schedule is replaced with the solution in DDPM. Can the authors further explain the novelty of proposed Domain Shift scheme? 2. The experimental setting of the paper is insufficient. See questions for details.

Questions

1. The method proposed in the paper is based on the RealESRGAN preprocessing and LAControlNet (from DiffBIR). Is the model trained based on the pre-trained model of DiffBIR? What about the results on the baseline1 "RealESRGAN preprocessing + DiffBIR" or baseline2 "RealESRGAN preprocessing + w/o Domian Shift Guidance(DoSG)"? Are there some experiments to verify the performance improvement of the proposed method compared to the baseline1 or baseline2? 2. What are the advantages of the DoSSR Solvers proposed in the paper compared to existing accelerated sampling schemes (such as DDIM [1], Dpm-solver++ [2], EDM [3])? Are there any ablation studies to verify it? The relevant references are listed as follows, [1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020). \ [2] Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv preprint arXiv:2211.01095 (2022). \ [3] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in neural information processing systems 35 (2022).

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors fully explain the limitations and potential societal impact in this paper.

Summary

The existing diffusion-based image restoration models do not fully exploit the generative prior of the pre-trained diffusion models and produce high-quality images starting from Gaussian noise, leading to inefficiency. To address these issues, the paper a Domain Shift diffusion-based SR model (DoSSR) which can leverage the generative power of the pre-trained diffusion models and reconstruct super-resolved images based on the input corrupted images. It shows that the proposed method can effectively improve efficiency with few evaluation steps. Specifically, experiment results show that the proposed model achieves a slightly better performance on some benchmark datasets, including synthesized data and real-world data, in terms of various perceptual quality measurements.

Strengths

1. The paper clearly describes the limitations of the existing diffusion-based image restoration models, which are hot topics in image restoration based on Gaussian diffusion models. The structure of the paper is well-organized, so the whole paper is understandable and easily follow. 2. How to efficiently perform image restoration based on the diffusion models is a significant challenge. The paper demonstrates that it can achieve better performance using few evaluation steps, which is impressive. 3. The paper evaluates the proposed method on several benchmark datasets, which contain synthesized data and real-world data, and the ablation studies are comprehensive.

Weaknesses

1. In implementation, this research adopts LAControlNet and SD 2.1-base as the pre-trained T2I model. However, the paper does not illustrate the overall structure of their proposed method, so it is hard to understand how to adopt the corrupted input as conditional information. 2. The novelty of the paper is limited. 1). Leveraging pre-trained large-scale diffusion models for image super-resolution under the framework of ControlNet has been investigated in many previous studies [1-4]. These methods can also produce high-quality restored images from the corrupted input. 2). The goal of the proposed domain shift SDE is to design a diffusion bridge between the corrupted input images and the corresponding high-quality images, which is very similar to flow matching-based methods, such as rectified flow [5]. However, the paper does not discuss the flow-based models. 3. By leveraging flow-based models, several methods [6-7] have been proposed for image restoration. 4. The performance improvement on several datasets is limited. It is hard to evaluate the visual quality of images generated by the proposed method through visual comparison because no GT image is illustrated. [1] Wang J, Yue Z, Zhou S, Chan KC, Loy CC. Exploiting diffusion prior for real-world image super-resolution. IJCV, 2024. [2] Chen X, Tan J, Wang T, Zhang K, Luo W, Cao X. Towards real-world blind face restoration with generative diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology. 2024 Apr 1. [3] Yu F, Gu J, Li Z, Hu J, Kong X, Wang X, He J, Qiao Y, Dong C. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR 2024. [4] Yang T, Ren P, Xie X, Zhang L. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308.14469. 2023. [5] Liu X, Gong C, Liu Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023. [6] Liu J, Wang Q, Fan H, Wang Y, Tang Y, Qu L. Residual denoising diffusion models. CVPR, 2024. [7] Zhu Y, Zhao W, Li A, Tang Y, Zhou J, Lu J. FlowIE: Efficient Image Enhancement via Rectified Flow. CVPR, 2024.

Questions

1. It is better to illustrate the overall structure of the adopted models. For example, how to adopt the corrupted input as conditional information for the T2I model. 2. The novelty of the paper is a significant concern. It is better to demonstrate the advances of the proposed method, compared with the previous studies. 3. The paper proposes to specifically design the drift term in SDE for image super-resolution, which leads to the reduction of evaluation steps. It easily causes confusion, because the additional drift term is to interpolate the corrupted input and the corresponding high-quality images. The sampling method is the key point. These two points should be clarified.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The paper does not discuss the limitations and social impact of the proposed method. It is hard to evaluate the model fairness, generalization ability, robustness, and transparency of the proposed method.

Summary

This paper proposes a Domain Shift diffusion-based SR model, named DoSSR, that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. Introducing the SDEs to describe the process of domain shift and provide an exact solution for the corresponding reverse-time SDEs. Designing customized fast samplers, resulting in even higher efficiency, thereby achieving the state-of-the-art efficiency performance trade-off.

Strengths

1.This paper is innovative and combines a domain shift equation with the existing diffusion model to obtain better inference results and efficiency. 2.This paper has a solid theoretical foundation, the author gives a strict and sufficient formula derivation process. 3.The fusion results obtained are very superior in the non-reference index.

Weaknesses

1. In Table 1, the results of the proposed method on the reference index seem unsatisfactory, even much lower than other diffusion model methods. 2. In colddiff [r1], Bansal et al proposed a method of reasoning from an arbitrary starting point. What is the difference between the method of reasoning from LR mentioned in this article and it? Such clarification would be useful. 3. In order to facilitate observation, it is recommended to perform a partial zoom on Fig. 4 (b) for closer examination. [r1] Bansal, Arpit, et al. "Cold diffusion: Inverting arbitrary image transforms without noise." Advances in Neural Information Processing Systems 36 (2024).

Questions

1. As mentioned in Weakness, why do the resulting graphs look very beautiful, but the reference indicators are so far behind? 2. Explain the differences with DPM-solver, since ito-taylor expansionin also exists in your paper.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper presents DoSSR, a novel framework for diffusion-based image super-resolution that aims to balance efficiency and performance. A domain shift equation that integrates with existing diffusion models, allowing inference to start from low-resolution images while leveraging pretrained diffusion priors. A continuous formulation of the domain shift process using stochastic differential equations (SDEs), enabling fast and customized solvers.

Strengths

1. The domain shift equation and SDE formulation provide a solution and enables efficient inference. 2. DoSSR outperforms existing methods on multiple benchmarks, both in terms of quality metrics and computational efficiency.

Weaknesses

I have doubts about the interpolation between LR and HR. Theoretically, from the moment of interpolating, the image contains both the degradation of LR and the details of HR. For details, please refer to the article [Interpreting Super-Resolution Networks with Local Attribution Maps], which also mentions similar gradient and difference operations. If each image contains information about LR and HR, is it reasonable to use this for training? Based on the proposed method and the actual effect of LR&HR, I guess that using only LR or adding various noise/blur methods to LR may also achieve similar results. Since the topic of the paper is related to this approach, this concept is still important. In addition, skipping the first half of the diffusion model is a more intuitive method. I don’t know if there is a paper discussing this, but in practice this method seems to have appeared for some time (this will not affect my score much).

Questions

About the issue mentioned in Weaknesses, can we have some experiments to show the reason of doing such interpolation?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Follow-up Feedback

Thank you for your respond. However, I still have some concerns about the real application value of the proposed DoS SDE-Solver. As the author said, "Our proposed sampler, DoS SDE-Solver, is specifically designed for the model under our forward-noise scheme framework" and "Existing samplers, such as DDIM, DPM-Solver, and EDM ... cannot be adapted to our model". Therefore, I doubt whether DoS SDE-Solver can be easily extended to other models or tasks, and whether it will bring a large training cost.