Re-boosting Self-Collaboration Parallel Prompt GAN for Unsupervised Image Restoration

Deep learning methods have demonstrated state-of-the-art performance in image restoration, especially when trained on large-scale paired datasets. However, acquiring paired data in real-world scenarios poses a significant challenge. Unsupervised restoration approaches based on generative adversarial networks (GANs) offer a promising solution without requiring paired datasets. Yet, these GAN-based approaches struggle to surpass the performance of conventional unsupervised GAN-based frameworks without significantly modifying model structures or increasing the computational complexity. To address these issues, we propose a self-collaboration (SC) strategy for existing restoration models. This strategy utilizes information from the previous stage as feedback to guide subsequent stages, achieving significant performance improvement without increasing the framework’s inference complexity. The SC strategy comprises a prompt learning (PL) module and a restorer (<inline-formula><tex-math notation="LaTeX">$Res$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="ren-ieq1-3589606.gif"/></alternatives></inline-formula>). It iteratively replaces the previous less powerful fixed restorer <inline-formula><tex-math notation="LaTeX">$\overline{Res}$</tex-math><alternatives><mml:math><mml:mover><mml:mrow><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow><mml:mo>¯</mml:mo></mml:mover></mml:math><inline-graphic xlink:href="ren-ieq2-3589606.gif"/></alternatives></inline-formula> in the PL module with a more powerful <inline-formula><tex-math notation="LaTeX">$Res$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="ren-ieq3-3589606.gif"/></alternatives></inline-formula>. The enhanced PL module generates better pseudo-degraded/clean image pairs, leading to a more powerful <inline-formula><tex-math notation="LaTeX">$Res$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="ren-ieq4-3589606.gif"/></alternatives></inline-formula> for the next iteration. Our SC can significantly improve the <inline-formula><tex-math notation="LaTeX">$Res$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="ren-ieq5-3589606.gif"/></alternatives></inline-formula> ’s performance by over 1.5 dB without adding extra parameters or computational complexity during inference. Meanwhile, existing self-ensemble (SE) and our SC strategies enhance the performance of pre-trained restorers from different perspectives. As SE increases computational complexity during inference, we propose a re-boosting module to the SC (Reb-SC) to improve the SC strategy further by incorporating SE into SC without increasing inference time. This approach further enhances the restorer’s performance by approximately 0.3 dB. Additionally, we present a baseline framework that includes parallel generative adversarial branches with complementary “self-synthesis” and “unpaired-synthesis” constraints, ensuring the effectiveness of the training framework. Extensive experimental results on restoration tasks demonstrate that the proposed model performs favorably against existing state-of-the-art unsupervised restoration methods.