Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction

Summary

This paper proposes a fast test-time adaptation SR framework named SRTTA, which is capable of super-resolving images with various degradation types while maintaining efficiency. Specifically, the authors first exploit a pre-trained degradation classifier to predict the degradation type of the test image and construct the paired dataset based on the degradation. Then, the authors adapt the SR model by conducting the feature-level reconstruction learning from the test image and its second-order degraded image generated by the designed second-order degradation scheme. Finally, the authors design the second-order reconstruction loss to adapt the pre-trained model to test images with different degradation types. After reviewing the rebuttal, I've noticed that most of my previous concerns have been addressed. Therefore, I would like to increase my initial score.

Strengths

This paper provides an interesting image super-resolution idea for different degradations in the real-world test-time scenario. Compared with the classical deep network, it is applicable to any pretrained SR model. The proposed SRTTA can adapt a given SR model to estimate the degradation shifts in the test images. In addition, the authors construct a dataset named DIV2K-C with 8 different degradation types.

Weaknesses

1. The proposed framework seems to be oriented towards a single degradation from different degradation types in the real-world test-time scenario. However, for the real-world scenario, the practical degradation is much more complex and could be the combinations of different degradations. This paper may not be applicable to this situation. 2. The proposed dataset also may not be suitable for the aforementioned situation. 3. The methods for visualization comparison are older compared to the methods for quantitative comparison. 4. The GPU Time of the proposed method in Table 1 is not the optimal. 5. The notation in section Super-Resolution Test-Time Adaptation should be consistent and continuous, such as x_t in line 149.

Questions

1. The authors should clarify the 1st and 2nd concerns in section weaknesses. This seems to be an important reason why the proposed method is superior to other SOTA methods. 2. A detailed explanation or citation for the domain shifts should be given in line 136 on page 3. 3. The authors should clarify the 3rd concern in section weaknesses and provide more convincing visualization. 4. The authors mention that some previous methods are quite time-consuming in the motivation, so the authors should clarify the 4th concern in section weaknesses. 5. How does the proposed network work when the test images are degraded by multiple degradation types at the same time? 6. The motivation mentioned from line 158 to line 160 in section 4.1 should be further demonstrated or cited.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The application scenario of the proposed method is limited for the single degradation.

Summary

The paper proposes a test-time adaptation framework called SRTTA for image super-resolution (SR) to address the degradation shift issue between training and test images. The SRTTA framework adapts a pre-trained SR model to different degradation types observed in real-world test images. It introduces a second-order degradation scheme that generates paired data based on the degradation type predicted by a pre-trained degradation classifier. The SR model is then adapted through feature-level reconstruction learning from the initial test image to its second-order degraded counterparts. Experimental results on a newly synthesized DIV2K-C dataset and a real-world DPED dataset demonstrate the effectiveness and practicality of the SRTTA framework.

Strengths

The paper introduces a new framework, SRTTA, that allows rapid adaptation of a pre-trained SR model to various degradation types observed in test images. The second-order degradation scheme and feature-level reconstruction learning provide a practical and effective solution for adapting the model to real-world scenarios. Unlike some existing methods that focus on a single degradation type, SRTTA considers multiple degradation types, such as blur, noise, and JPEG artifacts, making it more applicable in real-world scenarios where images can have diverse degradation types. SRTTA achieves fast adaptation by using only a single degraded low-resolution image during testing. This makes it more practical for real-time applications compared to methods that require extensive iterations or GAN-based approaches for degradation estimation.

Weaknesses

1. Although the paper claims to evaluate the framework on a real-world DPED dataset, the evaluation results and analysis are not sufficient, in the per. The lack of comprehensive evaluation on diverse real-world datasets limits the generalizability and applicability of the proposed framework. 2. The paper focuses on adapting the pre-trained SR model to different degradation types using second-order degradation. However, there is limited discussion or analysis on the generalization capability of the adapted model to unseen degradation types. It would be valuable to investigate the model's performance when faced with degradation types not encountered during training, or different types combine the degradations. 3. The paper primarily focuses on evaluating the performance of the proposed method in terms of PSNR (Peak Signal-to-Noise Ratio). While PSNR is a commonly used metric, it does not always correlate well with perceived image quality. The evaluation could have included other quality metrics, such as Mean Opinion Score (MOS) or other human eye-related metrics, to provide a more comprehensive assessment of the proposed method.

Questions

refer to the weakness

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

yes, the authors have addressed some limitations of the paper

Summary

This paper proposed a fast test time adaption method for image SR. The authors point out that the existing image SR method use synthesized paired LR-HR data for training, which degradation type has a gap with the real-world sceneries. The authors also proposed a second-order based data synthesis circuit with a degradation classifier. The proposed method achieves impressive results on both real-world and synthesis data. After reading the author's response, most of my problems have been addressed. Therefore, I would like to increase my initial score.

Strengths

Strengths: Experimental results show that the proposed method is effective for real-world image super-resolution. Visual results are clear to show the improvement.

Weaknesses

Weaknesses: 1. The concept of second-order degradations is already proposed by real-ESRGAN in 2021. 2. Please see the following detailed comments and suggestions.

Questions

Here are some detailed comments and suggestions for improving the quality of this paper： 1. In the abstract, I think it’s better to say ‘conventional SR’ or ‘PSNR/fidelity oriented SR’use paired data for training. 2. Novelty problems, second order degradation is already proposed by real-ESRGAN, the authors need to enhance the explanation of the contribution of this paper. 3. It is better to give a definition of the second-order degraded image like Equation.1 4. The degradation category definition is a discrete single label, second order requires repetition and mixing, so is the design of the classifier reasonable? Did the authors consider the sequence of the degradations? 5. The diagonal Fisher Information matrix in Section 4.3 requires independent identical distribution, is the image identically distributed under different degenerations? Equation 7 only reflects the stability of the same degradation under different augmentations, which does not seem to be able to target different degradations. In Equation 8, the parameter selection part I think is also not reasonable enough, through the artificial design of hyperparameters is usually difficult to meet the different datasets, so it is recommended to let the parameter be learnable. 6. The visualization results look a bit over-smoothed, it is suggested to add some analysis to it.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the above comments.

Summary

The submission presents Test-time adaptation Super Resolution (SRTTA) via learning from the second-order degradation and reconstruction. This method super-resolve images with various degradation types while maintaining high efficiency. In addition, the paper also introduce a new dataset DIV2K-C, C stands for corruption, to better depict various image corruptions in the real-world SR use cases. The method is compared to other SOTA based this new dataset and achieves better metrics.

Strengths

There are 2 major contributions in this submission: 1) Adaptive Parameter Preservation module for the test time adaptation; 2) DIV2k-C dataset.

Weaknesses

I'm really doubt about the problem setting. First, modern ISP always removes noises like Poisson noise, most images are clean besides some compression artifacts. Second, as the authors admit, if an image has some corruptions, it usually has more than 1 type of corruptions. The submitted method based on a classifier to choose at most 1 types of corruption does not follow the real-world scenarios. Both of these two weakness challenges the fundamental design of the method. In addition, the method is only evaluated on the new dataset created by the authors, which is far from enough.

Questions

Since your degradation classifier is to determine the types of the degradation, which is usually a local effect, what's the reason of putting the whole image into the classifier (I assume you also have to resize your input image to 224x224) instead of putting patches of your input image into the classifier?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The paper proposes to solve the domain shift problem for a pretrained super-resolution model. A pretrained classifier is utilized to predict the type of degradation of the test image. After the degradation type is detected, the degradation is used as an augmentation scheme for adaptation training for the new SR model.

Strengths

The paper is easy to follow. The motivation for the work is reasonable since super-resolution is an important task in Computer Vision, while the existing methods only focus on one degradation type. This results in the poor performance of the SR model in some cases.

Weaknesses

The analysis is the weakest part of the work: 1. What are the consequences of domain shift when applying pretrained SR models? The work only mentions blur degradation is the most popular degradation type that other SR models concentrate on. However, no qualitative or quantitative observation is provided if another type of degradation happens in the dataset. What happens if the provided test datasets are degraded by blur and JPEG degradation? or both blur and noise degrade the provided test datasets? or only JPEG, or only noise? 2. In practice, which type of degradation happens the most? Whether the cover for the domain shift is necessary? Which scenarios will we need to do the domain adaptation training for the SR model? 3. The training of classifier in C.1 section mentions 8 types of degradation. What are the 8 types? Why, in the main paper, the classifier only has 4 classes? 4. The qualitative results in Figure 3 shows that KernelGAN+ZSSR is better than the proposed method. 5. In line 185, the author mentions, "encourage the pre-trained model to learn how to reconstruct ...", so the pretrained model will also be trained?

Questions

see the weaknesses (Weaknesses are addressed after rebuttal and discussions )

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

no limitations

Good Rebuttal (with additional minor suggestions)

Thank you for your response. After reviewing the rebuttal, I've noticed that most of my previous concerns have been addressed. Therefore, I would like to increase my score. However, I still have a few more suggestions: The details of the SRTTA-MC should be included in the revised paper. Given that the inference speed of the proposed SRTTA isn't the fastest, it would be beneficial to include the analysis mentioned in A4 within the experiments. This could help clarify the reasons behind this aspect. The hyperparameter $\rho$ should be explicitly incorporated into Equation (8). While the authors investigate the sensitivity of $\rho$ in Equation (8) in Table 4, the absence of a direct reference to $\rho$ within Equation (8) could potentially hinder a clear understanding of the relationship between the hyperparameter $\rho$ and the equation.

the authors provide comprehensive responses

The authors provide comprehensive responses to the raised concerns, reaffirming the efficacy of their SRTTA method in handling real-world degradations, adapting to unseen degradation types, and producing images with improved visual quality.

Reviewer-author discussion

Many thanks to the reviewers who have provided feedback to the authors. Dear @Reviewer tgyp, @Reviewer pkZJ, and @Reviewer aSmb, We kindly request you to take a moment to review the authors' responses and share your opinions or any remaining concerns at your earliest convenience. Your input is greatly appreciated. Thank you. AC

How did you input the whole image into the classifier?

Up to now, I still don't buy the point of inputting the whole image into your ResNet-50, especially, you train this classifier with only 224x224 patches, as stated in your supplemental material C.1. The default input size of ResNet-50 is definitely 224x224, and your training makes this model also only learn the pixel relationships from that particular input size. It means the 7x7 and 3x3 kernels in this classifier are only familiar with pixel relationships in that resolution. How can it work on images with arbitrary resolutions, say 1080p, 1920x1080.