RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

Summary

For real-world images corrupted by multiple simultaneous degradations, this paper first analyzes the limitations of using all-in-one restoration models and various task-specific models. The authors then introduce RestoreAgent, which automatically identifies the types of degradation in a degraded image, determines the sequence of restoration tasks, and selects suitable models from the model pool. RestoreAgent presents an automated restoration pipeline that requires only an input image and a general human instruction, without any prior knowledge of the involved degradation tasks or manually predefined task sequences.

Strengths

1. The paper comprehensively analyzes the challenges and limitations of employing all-in-one models and multiple task-specific expert models with fixed or random task sequences, as well as fixed or random models for each task. 2. The authors evaluate various configurations of RestoreAgent using diverse objective image quality metrics (PSNR, SSIM, LPIPS, DISTS, and their combinations), all of which outperform the human expert model on the corresponding metric. 3. RestoreAgent exhibits the scalability by extending to new tasks and models with minimal computational resource. 4. The presentation, including writing, analysis, and visualization, is clear and easy to follow.

Weaknesses

1. Incomplete descriptions about data construction. - Authors randomly select up to four types of degradation from a degradation set (noise, blur, JPEG, rain, haze, and low-light) to construct paired training data. According to data synthesis strategies in [1,2], JPEG compression is typically performed after noise and blur, and in the final order. Is the degradation order of JPEG compression in this paper the same? If not, the authors should discuss the reasonableness of random sampling. - What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image? - What is the configuration in ablation studies about training data amount? Simultaneously scaling up low & high-quality images or synthesizing more low-quality images for each high-quality image? If it’s the former, will increasing the number of degraded images while keeping the number of high-quality images unchanged improve performance? 2. Inference time for input images with diverse resolution. - The authors are suggested to report the running time for input images of various resolutions. This should include the total time, the running time for the RestoreAgent, and the running time for the subsequent restoration models. The reviewer is curious whether the agent's response time exceeds that of the restoration models when processing high-resolution images, such as those with 4K resolution. 3. Scalability for new tasks and models. - Section 4.5 demonstrates that the proposed RestoreAgent can extend to new tasks and models in just half an hour, surpassing human expert-level performance on the new task. However, it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning. The authors are encouraged to report the performance of the fine-tuned model on the previous tasks to address this concern. [1] Wang X, Xie L, Dong C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1905-1914. [2] Zhang K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4791-4800.

Questions

Addressing concerns in the Weaknesses with thorough explanations and additional experiments would significantly enhance my confidence in this work. A satisfactory response to these points may lead to a reconsideration of the current evaluation.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The manuscript includes the checklist guidelines.

Summary

This paper proposes a new pipeline to address multiple degradation, like noise, blur and low light. Besides, a RestoreAgent with multimodal large language models is introduced to assess the type and extent of degradations in the input images and perform dynamic restorations.

Strengths

1. The paper is well-written and well-organised. 2. The whole pipeline seems to be novel and reasonable. 3. The method achieves SOTA performance on several benchmarks and different degradation tasks.

Weaknesses

The overall motivation of this paper is commendable, but I have a few concerns: 1. The author mentions that RestoreAgent can autonomously assess the type and extent of degradation in input images and perform restoration. This strategy is interesting. However, I am wondering how the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance? 2. In contrast to other image enhancement techniques, the proposed RestoreAgent should first find a suitable restoration task and then select the most appropriate model to enhance the quality of the input. Therefore, I am concerned whether this process will increase the inference time. The authors should provide some computational analysis. 3. The enhancement capabilities of this work rely heavily on existing enhancement frameworks. If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section. 4. The explanation of "ranking" and "balanced" in Table 1 is still unclear. The authors should clarify the definitions of these terms. 5. It would be better to show more visual comparisons of the RestoreAgent.

Questions

Please see weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are not mentioned in the paper.

Summary

This paper presents an image restoration pipeline designed to handle various degradation types and levels by leveraging MLLM’s capabilities to select the appropriate model and determine the execution order. It begins with an analysis of why execution order and utilizing multiple models for different degradation levels are crucial for restoring complexly degraded images. The paper then constructs an instruction dataset and fine-tunes the MLLM. Experimental results demonstrate the effectiveness of the proposed restoration pipeline.

Strengths

1. This work presents a compelling analysis of complex image restoration. This insight is valuable given that degraded images in real-world scenarios often involve multiple types of degradation. 2. This approach leverages the strengths of different models for handling specific noise levels, thereby eliminating the trade-off between generalization and performance. 3. This paper formally defines the problem of handling multiple degradations and model selection in image restoration. 4. Extensive experiments demonstrate superiority of such pipeline in processing degraded images with multiple degradations.

Weaknesses

1. In the introduction, it would be helpful to explain how the Multi-Level Learning Model (MLLM) excels at understanding different types and levels of image degradation. This will show why MLLM is well-suited for handling complex combinations of image degradation. Providing this clarity will make the benefits of using MLLM for image restoration more evident. 2. When incorporating a new type of degradation, the cost extends beyond merely training the MLLM. Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data. 3. In lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript "i" is already used for degradation type and it might be clearer to use another character.

Questions

1. What if the degradation of the input image falls outside the predefined degradation scope? This could present a generalization issue, as the model might not perform well on unseen types or levels of degradation not covered in the predefined scope. Please discuss it. 2. In Table 2, it would be clearer to highlight the best method for each evaluation criterion. Additionally, please specify which methods the ranking improvement is compared against for better context.

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations.

Summary

This paper introduces RestoreAgent, an innovative image restoration system that leverages multimodal large language models to autonomously handle images with multiple types of degradation. The system addresses limitations of existing all-in-one models and fixed task sequences by dynamically adapting to each image's specific degradations. RestoreAgent can identify degradation types, determine appropriate restoration tasks, optimize the execution sequence, select the most suitable models, and execute the restoration process autonomously. The authors present a method for constructing training data and demonstrate that RestoreAgent outperforms existing methods and human experts in handling complex image degradations.

Strengths

1. This paper represents a innovation and a good contribution in image restoration and potentially opens up a new research direction for this area. 2. The motivation is strong. The authors effectively demonstrate the importance of task execution order and model selection in multi-task scenarios. The designed system adeptly addresses these issues. 3. Experimental results indicate that RestoreAgent's decision-making capabilities in handling complex degradations surpass those of human experts. This kind of pipeline also surpass all-in-one models. 4. The paper is generally well written and clear to understand.

Weaknesses

1. The paper constructs a training dataset for training the multimodal large language model and a testing dataset as a benchmark for evaluating performance across multiple tasks. More details and explanations regarding the construction methods of these datasets would be beneficial. 2. Table 1 presents performance rankings using both ordinal and percentage forms. The definitions and explanations for these ranking forms are somewhat lacking, which might require readers to spend extra time understanding them. Clearer explanations would facilitate better comprehension. 3. The proposed Autonomous Restoration Agent represents a novel paradigm that is likely to encounter numerous new challenges. Beyond the issues already mentioned in the paper, the authors could consider discussing additional limitations and future research directions for this paradigm. This would help future researchers better follow and improve upon this work.

Questions

1. The current method appears to predict all execution steps at once for a given input image. In Figure 3, each image has a dashed line pointing to the input. Does this imply that after each execution, the result can be fed back as input? (Based on my understanding, this system supports this) The paper seems to lack analysis and experiments related to this aspect. Could the authors provide more details on this part? 2. The authors have proposed a testing dataset to evaluate multi-task processing capabilities. Will this dataset be made publicly available to facilitate further research by other researchers?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See Weaknesses