ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting

Summary

This paper constructs a Markov chain that transfers between high-resolution and low-resolution by shifting the residual between them. They achieve competitive performance to SOTA methods with only 20 sampling steps.

Strengths

1. The paper structure is clear and the proposed method is well-motivated. 2. It only requires small iterating steps to achieve competitive performances.

Weaknesses

1. A brief figure to introduce Resshift should be provided. 2. Why this method is called ResShift? The motivation and function should be discussed. Moreover, the ablation study of the res-shift should be provided.

Questions

1. Since there are many math equations, it may be difficult for other researchers to reproduce the method. Would the implementation code be released?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations.

Summary

This paper studies diffusion-based image super-resolution (SR) with the goal of reducing the number of diffusion steps. The key intuition is to only learn the residual between an LR-HR image pair, thereby shortening the diffusion path. To this end, the paper introduces ResShift, a novel diffusion framework where the mean of the prior distribution is the LR image. It further provides a derivation of the forward and reverse processes, identifies an effective parameterization of the denoiser network, and analyzes the fidelity-realism tradeoff induced by the noise schedule. Extensive quantitative and qualitative experiments are conducted to verify the effectiveness and efficiency of the proposed method. In particular, ResShift achieves competitive results on the 4x SR task at a low cost of 20 diffusion steps, outperforming the strong baseline of LDM under the same sampling budget.

Strengths

- The paper makes the key observation that the residual between an LR-HR image pair is often small. It builds a novel diffusion framework around this insight to reduce the number of sampling steps for improved efficiency. Incorporating domain knowledge into the design of a diffusion model is a key strength of the paper. - Within the proposed diffusion framework, the paper studies the fidelity-realism tradeoff induced by noise schedules through detailed theoretical analysis and ablation experiments. - The proposed method is evaluated on both synthetic and real datasets. It demonstrates comparable or sometimes better results with respect to strong baselines. In particular, it consistently outperforms LDM under the same sampling budget (20 or 40 steps). - Overall, the paper is well-written, and the flow of presentation is easy to follow.

Weaknesses

- My main concern is that the comparison with LDM is unfair. In the main experiments, the number of diffusion steps is limited to 20 or 40. This makes sense, as the goal is to demonstrate that ResShift yields stronger results under a tight sampling budget. However, LDM is put at a significant disadvantage in this comparison, because its default DDIM sampler is not well-suited for short sampling paths. In this regime, more powerful samplers exist (e.g., Heun and DPM solvers) and, according to the reviewer's own experience, will likely produce far better results within 20 or 40 NFEs. This raises an important question: is the proposed diffusion model tailored for SR truly superior to the vanilla model in terms of efficiency, when the latter is paired with a fast solver? This is key to assessing the paper's contribution, yet is not addressed by the current set of experiments. - DDRM is another diffusion-based model for SR in addition to LDM. The authors are encouraged to compare ResShift against DDRM as well. - The qualitative results on real data seem inconsistent across different images. In Figure 4, ResShift produces sharper and more detailed output than the baselines, whereas it is more prone to artifacts as shown in Figure 1 in the supplement. I am wondering what causes this discrepancy. Are the images from different real datasets, or is there just a lot of randomness?

Questions

Please see the section above for questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses limitations of the method. The authors are encouraged to provide a short discussion on the potential negative societal impact of their work, such as hallucination of inappropriate image content.

Summary

The paper presents a new image super-resolution diffusion model called "resshift", which aims to address the efficiency issue in diffusion models. Existing acceleration strategies often yield over-smooth results. To combat this, the authors propose a new diffusion model for super-resolution that can produce favorable results in just 20 sampling steps. This method is based on operating on residuals and the paper also proposes a noise-adding strategy to aid this process. The main experiments are performed on real super-resolution settings, imitating the prior setup of Real ESRGAN, and includes testing on real images.

Strengths

The experimental results appear quite promising. Specifically, the outcomes produced with 20 steps seem slightly better than existing methods. This shows that the proposed method can efficiently achieve super-resolution with fewer sampling steps, and the introduced noise-adding strategy appears to be a beneficial supplement to the overall approach.

Weaknesses

The major issue with this paper lies in its presentation. The central idea of the paper is not clearly articulated. In the second section, although the method is described meticulously, it does not seem significantly different from existing methods. The authors do not clearly highlight what distinguishes their approach. It is recommended that the authors construct a table to illustrate why the "shift" and "residual" ideas are useful, and provide a conceptual narrative at the beginning of the second chapter to give readers an overall impression before diving into mathematical expressions. Secondly, Figure 2 could potentially be misleading. It should be the image obtained by adding noise in the VQ-VAE's latent space, but this is not just noise because a conversion to RGB image is performed. This needs further clarification since it's difficult to discern that Gaussian noise is added without any explanation. Lastly, the method's versatility needs to be elaborated on. The method does not seem to be confined to super-resolution and appears to be applicable to all image restoration models. The question arises whether it could be applied to other image processing tasks. There also needs to be more comparative analysis, as many recent acceleration methods, including CM models, have not been included in the comparison.

Questions

see Weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

Yes

Summary

The authors propose a new diffusion model, based on the principle of residual shifting, that is able to converge to a good looking image in a low number of diffusion steps, improving in terms of high resolution image inference time at least over the Latent Diffusion Model (LDM) SR variant, another well-established diffusion based solution for image upscaling. The authors provide extensive ablations (both is the main manuscript and the supplementary material) showing that the model is able to achieve a significant level of performance, at least in terms of subjective results analysis. The trade-off between the number of diffusion steps and the achieved "image quality" (subjective) is well explored. However, some concerns and comments will be found in the appropriate section of the review. The metrics chosen for performance quantification (PSNR, SSIM, LPIPS, CLIPIQA, MUSIQ) show mixed results, with the model characterized by SOTA performance in terms of non-reference image metrics, with strong indicators in terms of CLIPQA and MUSIQ. In terms of LPIPS, the model also shows an advantage over the compared methods. In terms of PSNR and SSIM, the model shows its limitations, however explainable, given the nature of the algorithm and the followed framework.

Strengths

1) The paper is generally well written and easy to follow. The provided details are enough to offer a good overview of the proposed method. 2) The authors provide ablations showcasing the influence of multiple parameters characterizing the proposed method. 3) The quality of the results selected for visual comparison matches the performance level in terms of quality assessment related metrics. 4) An advantage over the LDM in terms of inference time/diffusion steps needed to achieve a similar image quality assessment performance can be easily derived from the Table 2 of the main manuscript. The quality of the results provided for visual comparison and the perspective for more efficient diffusion models explain my rating.

Weaknesses

1) The efficiency claim made in the title is not clearly supported by the provided evaluations. Even if the advantage over LDM in terms of inference time and diffusion steps needed is clear, there is no comparison with the other methods considered for comparison (Table 3 and 4). This work should clarify the question mark over the opportunity of using a diffusion model for Efficient Image Super Resolution, show the model strengths and let the readers evaluate the trade-off between reconstruction fidelity and subjective "image quality". This is why I added this as the first weak point of the authors claim. 2) The choice in presenting the results of their ablative study (Table 1) versus the against-SOTA (Table 3,4) is somehow strange. the configurations in terms of data are different making it impossible to quantify the advantage between their proposed models and different configurations of LDM. Also, a fully supervised Image Super Resolution CNN architecture could be also added, to better support the efficiency claim as performance gain per inference-time millisecond. 3) A certain trend can be observed in the first 5 lines of the Table 1, where the performance of the model in terms of reconstruction fidelity decreases with the number of diffusion steps followed. This can show an overfitting behaviour on some properties of the training set that are not aligned with the properties of the testing split. 3.1) Have the authors focused their efforts on strategies to supervise the training of their model in order to retain as much as possible from the reconstruction fidelity, while achieving significant subjective performance? 4) When setting the numbers of diffusion steps to 20, there seems to be a strong correlation between the parameter p value and the reconstruction fidelity of the model. However, the value for p chosen for the final configuration was 0.8. Was the choice made given the impact in performance in terms of CLIPQA/MUSIQ? Why? 5) A comparison with another referenced method claiming improved inference time for a diffusion-based algorithm (maybe also Reffusion) would be useful for the potential reader, to understand the advantages of the proposed method (reference 48 of the main manuscript).

Questions

1) Where is the difference in the number of parameters of the diffusion backbone against the LDM coming from?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper introduces a diffusion-based image super-resolution method. The proposed method starts to generate an HR image from a given LR image directly (learns to generate the residual image between LR and HR), not start from a noise. Therefore, the method can generate the output image faster than previous works, specifically, with only 20 sampling steps. In addition, there are some hyperparameters to control the fidelity-perceptual quality trade-off of output images, namely, T, p, and k.

Strengths

- The idea of starting from LR image and utilizing a residual image is proper to the SR task and this is different from conditioning the LR image. - This successfully makes the method takes only short steps in the inference using a comparable number of parameters and runtime compared to previous method LDM. - In addition, it is practical for users to control the fidelity-perceptual quality trade-off of the SR model by adjusting the hyper-parameters. - The paper shows the superiority in terms of perceptual quality (LPIPS, CLIPIQA, MUSIQ) in synthetically degraded images and real images as well.

Weaknesses

- There is no visual comparison on synthetic SR testsets so it is hard to fully understand the effectiveness of the method, but the paper argues superiority in this aspect. - Several previous methods are listed in Sec 3 (Related Work), however, only LDM is compared in experiments. Are there any difficulties or problems in comparing with other methods?

Questions

- In L87, the image value range of [0,1] but I think the residual image is within [-1,1]. Does the following explanation still hold if I am correct? - In L135, how this is achieved? I think it is weird because the method does not learn to generate a noise distribution, and what does this description mean? - It would be better if runtimes/parameters are added for other methods in Table 3 (especially vs GAN-based methods).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Please see the weaknesses and limitations. - My rating is a reflection of the weaknesses and I look forward to the authors' feedback on the weaknesses.

Look forward to your feedback.

Dear AC and Reviewers, Thanks again for the constructive comments of all the Reviewers. As the deadline for discussion is approaching, we sincerely look forward to your further feedback. Please feel free to let us know if you have any further concerns or comments. Thanks. The authors.

Updated Rating

Since some of the concerns were addressed by the authors in their rebuttal, there is a clear advantage of the method against the compared LDM in terms of performance, while still lagging behind other GAN based approaches. Authors are encouraged to clearly describe their residual shift procedure, to emphasize their contribution, and also the novelty of their method. The evaluations provided in the rebuttal clearly would help the reader to understand the advantages of their method and its potential. Thus, I am considering a "Weak accept" rating for the submission, after the rebuttal.