One-Step Effective Diffusion Network for Real-World Image Super-Resolution

Summary

In this paper, authors propose a novel one-step effective diffusion network, termed as OSEDiff, for the Real-ISR. The proposed OSEDiff adopts LQ image as the input and directly output the final output with the help of the decoder of VAE, thus eliminate the uncertainty introduced by random noise sampling and achieve one-step generation of diffusion model. Besides, the variational score distillation is introduced in the latent space to conduct KL-divergence regularization to enhance the quality of final Real-ISR output. In the experimental part, the proposed method outperforms other baselines in both quantitative and qualitative comparisons.

Strengths

There are several strengths here: 1. One-step diffusion model is attractive and intereting, which will significantly reduce the inference time. 2. The paper is easy to read and understand. 3. The proposed method show comparable results in the experimental part.

Weaknesses

There are several weakness here: 1. Experimental part could be further improved, which will be further demonstrated in the following section. 2. The efficient and params of the proposed method is expected to be detailed demonstrated.

Questions

There are several concerns and suggestions here: 1. I am wondering the comparisons between the proposed method and SinSR. Does the improvement of performance gain from the pre-trained SD? 2. Experiments of the paper is expected to be improved. For instance, total parameters instead of the training parameters are expected to be compared in the Table 2, which is more closely related to the deployment. 3. I am wondering the reason why the setting of "w/o VSD loss" may achieve better PSNR than proposed method. I would be better to provide more visual results of it to demonstrate the effectiveness of the VSD loss.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Could be referred to above.

Summary

This paper introduces a diffusion-based real-world image super-resolution method, OSEDiff, which can efficiently generate high-quality images in just one diffusion step. Firstly, in order to eliminate the uncertainty introduced by random noise sampling, the authors propose to directly feed the low-quality images without any random noise into the pre-trained SD model which integrates trainable LoRA layers. Furthermore, the variational score distillation for KL-divergence regularization is utilized to align the distribution of generated images with natural image prior, ensuring that the one-step diffusion model could still generate high-quality output. OSEDiff achieves comparable performance to state-of-the-art SD-based real-world image super-resolution methods with fewer trainable parameters and inference time.

Strengths

This paper introduces a one-step effective diffusion network for real-world image super-resolution. The low-quality images without any random noise are utilized as the inputs of the pre-trained SD model and the VSD in latent space are utilized to align the distribution of generated images with natural image prior. The proposed method solves two problems in diffusion-based real-world image super-resolution and it achieves comparable performance to state-of-the-art SD-based real-world image super-resolution methods with fewer trainable parameters and inference time. The figures of this paper are simple and clear, with precise formula expressions, demonstrating a certain clarity.

Weaknesses

(1) The motivation is unclear. In the introduction, the authors use a certain amount of space to mention two major issues in training a Real-ISR model, including the problem of how to build the LQ-HQ training image pairs, but this paper does not propose an innovative solution to this problem. (2) The explanation of the novelty and its effectiveness are insufficient. 1) Contribution 1, "directly feeding the LQ images into the pre-trained SD model without introducing any random noise," claims to eliminate the uncertainty of output but lacks supporting evidence, raising doubts about its validity. It is recommended to provide visual evidence and analyze why this can eliminate the uncertainty and achieve better Real-ISR performance in detail. 2) Contribution 2, “utilizing variational score distillation for KL-divergence regularization,” it is recommended to provide some visualization of the output of diffusion network which are sent to the two regularizer networks to prove that it can align the distribution of generated images with natural image prior. (3) The metrics used need more explanation. Since CLIPIQA is text-related, although it has been proved to be effective and generalized in image quality assessment, it is not that convincing to directly apply it to evaluate the quality of real-world super-resolution images. More explanation can be provided. (4) Details of the network structure are lacking in the figure. In Figure 2, the detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers are lacking. It is recommended to provide detailed network structure. (5) In “Ablation Study”, the super-resolution performance of the method with text prompt extractor is not competitive enough compared to removing text prompt. Although the qualitative comparisons show that the method with text prompt extractor can generate richer image details, the method without text prompt is much better for several common metrics.

Questions

(1) Could you please provide some visualization of the output of diffusion network which are sent to the two regularizer networks to further prove the effectiveness of the regularization loss in the latent space? (2) Could you please explain in detail why "directly feeding the LQ images into the pre-trained SD model" can eliminate the uncertainty of output and achieve better Real-ISR performance than the methods which uses inputs with random noise? (3) Could you please provide detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers? (4) Could you please provide more qualitative comparisons of the method with/without text prompt extractor to further prove the effectiveness of text prompt or further elaborate on the importance of the text prompt to prove that it is essential?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

(1) Briefly explain why "directly feeding the LQ images into the pre-trained SD model" can eliminate the uncertainty of output, and why it can achieve better Real-ISR performance than the methods which uses inputs with random noise. (2) Please provide some visualization of the output of diffusion network which are sent to the two regularizer networks.

Summary

This paper presents a novel approach to real-world image super-resolution for a one-step effective diffusion network (OSEDiff). The proposed OSEDiff effectively eliminates the uncertainty introduced by random noise sampling in previous methods, achieving significant performance improvements across multiple benchmarks, demonstrating its potential for practical applications.

Strengths

1. The focus on one-step diffusion in this paper is particularly noteworthy, as it addresses a critical challenge for the practical application of diffusion models in image super-resolution tasks. 2. The performance gains and visually compelling results suggest the effectiveness of the proposed method for real-world image super-resolution. 3. The paper is well-structured and easy to follow.

Weaknesses

1. The core concept of this paper shares similarities with "One-step Diffusion with Distribution Matching Distillation". A comprehensive comparison with this work, including performance metrics, training efficiency (parameters and FLOPs), and potential limitations, would strengthen the paper's contribution. 2. While Table 2 provides a comparison of trainable parameters, including the total number of model parameters and FLOPs during inference would offer a more complete picture of computational efficiency. 3. There is a lack of discussion on the degradation in LQ images, such as how to ensure that text prompts remain accurate in low-quality scenarios. 4. In Figure 7 (first and second scenarios), some generated details and textures appear to be creatively added rather than strictly reconstructed from the LQ input. Addressing potential limitations in fidelity due to this creative aspect would be valuable.

Questions

No more questions.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed about the limitations.

Summary

The paper introduces OSEDiff, a one-step diffusion network for Real-World Image Super-Resolution. OSEDiff performs variational score distillation in the latent space to ensure predicted scores align with those of multi-step pre-trained models. ​ By fine-tuning the pre-trained diffusion network with trainable LoRA layers, OSEDiff can adapt to real-world image degradations. Experimental results show that OSEDiff achieves comparable or superior Real-ISR outcomes to previous methods. ​ The paper's contributions include the development of OSEDiff, which addresses the limitations of traditional multi-step diffusion models and enables efficient production of high-quality images in one diffusion step. ​

Strengths

1. The proposed method outperforms the previous single-step SR method (SinSR) in terms of the most of metrics. 2. It is comparable to previous multi-step diffusion models but greatly reduces inference time. 3. The model has very few training parameters and can be trained with minimal resources, which is advantageous. 4. Figure 3 shows that the qualitative comparison results are very promising.

Weaknesses

1. How much would the performance of OSEDiff change if multiple denoising steps were performed, such as 20 steps or 50 steps? 2. The design of the key module, the regularizer networks, is not clearly explained. 3. It is unclear how the LR image is used as a condition. Is it concatenated along the channel dimension in the latent space? However, the pre-trained SD’s VAE encoder downscales by a factor of eight, so for a 512x512 HR input, the latent output is only 64x64, which conflicts with the 128*128 LQ setting mentioned in the first paragraph of section 4.1. 4. The reason for fine-tuning the VAE encoder is not very clear. Please explain in the revision why it is necessary to fine-tune the VAE encoder jointly with the diffusion model. If this has already been discussed, please indicate where it is covered. 5. Does the model support input of LR images with different resolutions? For example, if the input is a 32x32 LR image, can feasible results be obtained?

Questions

Please refer to the weakness part

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Official Comment by Reviewer px3F

Thanks for the authors' rebuttal. Most of my concerns have been addressed. I still have one question about the SR results in Table 1. We can see from this table that 9 metrics are used for performance evaluation, and part of them are better than other methods. In my opinion, the metrics results of different models of the proposed method may vary significantly. Some models can be chosen to perform well on part of the metrics. In fact, the same applies to the comparison methods, i.e., they can also be tailored to specific metrics like this paper. This paper directly uses the default models of comparison methods, and haven’t chosen models for metrics like this paper. Could this approach potentially make some comparison methods perform better than the proposed method? This can be analyzed in the revised version.