Summary
The paper proposes a new framework for diffusion-based image restoration. The underlying architecture is based on that of "DvSR" [49], in which there is a deterministic predictor that does an initial restoration, used in conjunction with a probabilistic diffusion denoising network that applies the diffusion reverse process to estimate the additive residual to the initial prediction, that can be combined to get the final restoration.
The novelty is the change in the residual denoiser network (itself based on SR3 [42]). Instead of using spatially invariant convolutions that are fixed after learning, it employs the BPN [50] idea of having basis convolutional kernels that can be linearly combined at different spatial positions, with combination weights that are themselves estimated from a fusion of the initial deterministic restoration, and other auxiliary guidance (diffusion timestep and degradation type were mentioned).
In addition, a new inter-step patch-splitting procedure that is integrated with the diffusion denoising process is also used for patch-wise high resolution restoration, to overcome problems with artifacts at patch boundaries.
Experiments were conducted on the SID dataset [7] for low-light denoising, on the GoPro dataset [37] for motion deblurring, and on ImageNet JPEG compression artifact restoration.
Strengths
Generally there is decent novelty. Although the main ideas are taken from existing methods, the combination of these ideas are relatively new and not immediately obvious. The use of basis kernels from [50] for creating spatially-variant convolutional kernels for the diffusion denoiser is quite interesting.
The inter-step patch splitting, although simple, appears to be new and effective, with Fig. 6 quite convincing.
Experimental results substantially exceed current SOTA on perceptual metrics (LPIPS, NIQE, FID, KID), though more mediocre on distortion metrics (PSNR, SSIM). It is noted though that the DvSR network was retrained by the authors to match the same training hyperparameters that are more lightweight, due to inaccessible computational requirements of DvSR; otherwise the published results in [49] are better.
Weaknesses
On experiments:
- Since the method is essentially derived from DvSR, and the authors are retraining it based on their hyperparameters, it is important to conduct experiments with DvSR on the other low-light denoising and JPEG restoration tasks as well.
- Generalization tests from GoPro to the HIDE and RealBlur datasets have been done in various other baseline papers. The authors should have also tested on these.
- The NIQE numbers don't match the ones in [49], though the other perceptual metrics do. If they are different, the authors should properly justify in the paper why this is the case.
Multiple parts of the paper are unclear, as details have been left out.
- How is the auxiliary scalar information represented? For example, how are timesteps encoded? Details should at least be in the supplementary material.
- What exactly is happening in the inter-step patch-splitting procedure? Figure 2 is very vague. Although the results look good, and a reader can perhaps roughly guess what is happening, there are no details given.
- The descriptions in Section 4.7 Ablation Studies are very vague and need to be explained better.
- What happens in "w/o spatial guidance"? How does time and other scalar information get injected (or not)?
- What are "internal feature guidance", and "degraded image guidance"? These need clearer explanation.
The paper is littered with spelling errors. This can simply be resolved by running a spell-checker; that it is not done suggests poor neglect on the part of the authors. Some instances:
- L120: "Restoraion" => "Restoration"
- L210: "receipt" => "receptive"
- L257: "JEPG" => "JPEG"
Questions
Please answer the questions raised in the Weaknesses section.
The baseline methods are for the different restoration tasks are barely overlapping. How is the decision for the choice of baselines for each dataset? For example, Uformer is tested on both SID and GoPro, but not Restormer. Why aren't they be tested across the different tasks / datasets?
While it is generally accepted that spatially variant kernels can be useful in many situations, I think it is worth a discussion when it comes to denoising residuals. Intuitively we may expect the residuals have a more uniform spatial distribution, so kernels don't have to be spatially varying. This is unlike convolutions on normal images with strong spatially dependent image structures. But it seems the results suggest otherwise. Can the authors provide a better understanding on this?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The technical limitations given in Section 6 are quite reasonable.
However, the authors should also include some discussion about potential negative societal impact, since image restoration can also lead to hallucination of details that were not present in the original images.