DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation

Summary

This paper introduces a dual strategy for real-world image restoration: GenIR and DreamClear. GenIR, a novel data curation pipeline, addresses dataset limitations by creating a large-scale dataset of one million high-quality 2K images. DreamClear, a Diffusion Transformer-based model, utilizes generative priors from text-to-image diffusion models and multi-modal large language models for photorealistic restorations. It also incorporates the Mixture of Adaptive Modulator (MoAM) to handle various real-world degradations. Extensive experiments confirm DreamClear's superior performance, demonstrating the effectiveness of this approach.

Strengths

1.The writing and structure of the paper are clear and well-organized, making it easy to follow the authors' arguments and methodologies. 2.The authors' introduction of a privacy-safe dataset curation pipeline is significant. This is especially crucial in the era of large models, where data privacy and security are paramount concerns. 3.The experimental results convincingly demonstrate the potential of the proposed method, highlighting its effectiveness and applicability in real-world image restoration scenarios.

Weaknesses

1.The quality assessment of the newly constructed dataset is lacking, both quantitatively and qualitatively. Given the goal of real-world image restoration, I am concerned about whether the current pipeline effectively advances this objective. The authors need to include a discussion on this matter. 2.In the ablation studies, the authors should provide a detailed discussion on the interaction between the dual branches. The motivation for this design choice requires further explanation. 3.Real-world degradations are complex and intertwined. The authors should compare their approach with SUPIR using more challenging examples to better demonstrate the robustness and applicability of their method.

Questions

See the above weaknesses part.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the relevant limitations.

Summary

The paper introduces a dual strategy to tackle the challenges of image restoration (IR) datasets and the development of high-capacity models for image restoration. This strategy comprises: GenIR: An innovative data curation pipeline designed to bypass the laborious data crawling process, providing a privacy-safe solution for IR dataset construction. The authors generated a large-scale dataset of one million high-quality images. DreamClear: A Diffusion Transformer (DiT)-based image restoration model utilizing generative priors from text-to-image (T2I) diffusion models and the perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restorations.

Strengths

The authors made a significant contribution to creating a large and robust image restoration dataset while providing a privacy-safe solution. The authors demonstrated detailed evaluation with recent SOTAs. The restoration model shows good performance across different evaluation metrics. The model is robust enough to handle various degradations such as deblurring and denoising.

Weaknesses

For the created data, the paper claimed they maintained privacy standards to ensure no personal information was embedded in the generated images. However, the author failed to provide clear details on how these were achieved. Hence, it is difficult to establish the robustness and effectiveness of the privacy-preserving measures. Given that the SR model, DreamClear, is an integration of various restoration models and is also built on PixArt, a detailed comparative analysis of the computational complexity between the proposed framework and other existing methods would be beneficial. While DreamClear exhibits good performance in perceptual quality, its performance on traditional metrics like PSNR and SSIM is not as strong. Some grammatical errors are observed on line 96, and incomplete statements on lines 109 and 110 should be corrected.

Questions

Given that a million images were generated, how did the authors verify that none of the images had personal information?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes

Summary

The authors propose GenIR, a privacy-safe automated pipeline that generates a large-scale dataset of one million high-quality images for training of the image restoration (IR) models. Additionally, they introduce DreamClear, a IR model that seamlessly integrates degradation priors into diffusion-based IR models. DreamClear features the novel Mixture of Adaptive Modulator (MoAM), which adapts to diverse real-world degradations. Comprehensive experiments demonstrate its outstanding performance in managing complex real-world situations, marking a substantial progression in IR technology.

Strengths

The authors propose an automated data curation pipeline for image restoration and extensive experiments across both low-level and high-level benchmarks demonstrates the DreamClear’s state-of-the-art performance in handling intricate real-world scenarios.

Weaknesses

1. The overall impact of the generated dataset is not that convincing. Why is the training also performed on DIV2K, FLickr2K, LSDIR and Flickr8K. 2. The overall architecture in terms of contribution is too limited where the concept of mixture of experts modified to mixture of adaptive modulators and the text to image diffusion model have been widely already explored in the field of image restoration. 3. The first ablation study is not highlighting the impact of the proposed dataset. 4. It would be interesting to see the results without dual-based prompt learning if time permits?

Questions

1. Does the Eq. 2 and 3 holds only for the x_lq, does not it hold for x_ref?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes, the authors addressed the limitations.

Summary

The paper introduces "DreamClear," a high-capacity image restoration model, and "GenIR," an innovative data curation pipeline for image restoration (IR) tasks. DreamClear leverages Diffusion Transformer (DiT) models and generative priors from text-to-image (T2I) diffusion models, combined with multi-modal large language models (MLLMs), to achieve photorealistic restorations. GenIR is a dual-prompt learning pipeline that generates a large-scale dataset of high-quality images while ensuring privacy and copyright compliance. The combination of these strategies enhances the model's ability to handle diverse real-world degradations.

Strengths

1. GenIR provides a novel, privacy-safe method to create large-scale datasets, overcoming the limitations of existing datasets that are often small and not comprehensive. 2. DreamClear integrates generative priors from T2I diffusion models and employs a Mixture of Adaptive Modulator (MoAM) to handle diverse degradation scenarios effectively. 3. The model achieves high perceptual quality in restored images, compared with other methods in various benchmarks.

Weaknesses

1. Complexity in Implementation: The dual-branch structure and the integration of multiple experts for degradation handling add complexity to the model. 2. Some Technical details and more experiments should be provided.

Questions

1. Some technical details are not clear. For example, what kind of loss do you use in this paper? While the paper outlines the steps of the GenIR pipeline, it lacks specific details on parameter settings, such as the hyperparameters used in the generative models, data augmentation methods during training, and strategies for generating negative samples. There is insufficient detail on the criteria and standards used to evaluate and select the generated dataset. 2. One potential issue with the DreamClear approach is the risk of generating artificial textures during the image restoration process. Taking Figure 4 as an example, although the result is clearer, it may produce artifacts because the beak of this bird does not look like this. 3. In Figure 6, as data size increases, performance improves. Is there further improvement for more datasets, e.g., increasing the number of training images to 200000? 4. It would be better to compare the efficiency of different methods, e.g., model size, training/inference time, FLOPs. 5. Some necessary related work should be discussed. There are many diffusion-based image restoration methods, e.g., DDRM, DDNM, DeqIR, etc. It would be better to cite and discuss them.

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Please refer to the details above.

Request for Review of Rebuttal and Clarifications

Dear AC and Reviewers, Thank you for taking the time to review our paper. We are delighted to see that the reviewers recognized the strengths of our work. We are particularly grateful that Reviewer CpjF highlighted the novelty of our method, Reviewer FDew and Reviewer bAzb acknowledged the significant contribution it makes, and Reviewer A2Rj commended its state-of-the-art performance. However, the reviewers also raised some concerns. Reviewer CpjF sought more technical details about GenIR, while Reviewer A2Rj was interested in the overall impact of the generated dataset. We have provided a detailed rebuttal addressing these concerns and kindly ask the reviewers to review our responses when convenient. Thank you once again for your valuable feedback. Sincerely, Authors of Submission 8300

Thanks for the time and efforts invested by all reviewers and AC

Dear AC and Reviewers, Firstly, we wish to express our profound gratitude for your time, effort, and insightful feedback on our manuscript. We deeply appreciate the recognition our work has received: Reviewer CpjF emphasized the novelty of our method, Reviewer FDew and Reviewer bAzb acknowledged its significant contribution, and Reviewer A2Rj praised its state-of-the-art performance. Throughout the review process, several concerns were raised, primarily related to technical details, efficiency comparisons, and the necessity for additional ablations and comparisons. We have addressed most of these concerns during the rebuttal and discussion stages. A lingering concern pertains to the real-world applicability of our model, given its relatively high training cost. To address this, we highlight three points. **First, we've compared our method to the recent SOTA method SUPIR [CVPR 2024] .** DreamClear is less resource-intensive and needs at least **70 fewer A100 GPU days** than SUPIR. **Second, we emphasize the importance of our dataset and model considering the trend towards large models in fields like NLP and multimodal understanding.** These foundational models inherently need vast data and computational resources. In our paper, we tackle this issue from data and model viewpoints, aiming to push the boundaries of image restoration models. **Lastly, our generated dataset is valuable for real-world applications.** As shown in Figure 6 and our response to Reviewer A2Rj, our high-quality dataset, when compared to existing public datasets, can significantly improve the performance of lightweight models like SwinIR in real-world image restoration. We believe that the dataset and real-world image restoration model proposed in our paper are of significant value to the field of low-level vision and can effectively push forward the progress of the image restoration field in the era of large models. We sincerely appreciate the time and efforts invested by all reviewers and AC in evaluating our work. Best regards, Authors of Submission 8300