A Modular Conditional Diffusion Framework for Image Reconstruction

Summary

This paper proposes a modular conditional diffusion framework for image reconstruction. Specifically, a small module is trained and combined with pretrained IR networks and DPMs. Experiments show that this method is effective.

Strengths

1. According to the quantitative and qualitative results, the method shows great performances on several tasks. 2. The method can be applied to tasks combining task-specific pre-trained models, making the method easy to be extended. 3. The module requiring additional training is small and the method can be trained efficiently.

Weaknesses

- The figures in this paper are of poor quality. The authors should revise the figures to make the formulas clear and professional. - Overall I think the modular framework is of limited insight. The design is clean but not so informative for image restoration and some other related domains. - If I understand correctly, this method is agnostic to the specific tasks of image restoration, though the authors present several tasks in this paper, it would be good if more qualitative results on various tasks are given. - I think the arguements in line 45-49 are not proper. As the authors discuss the conditional space, sometimes we may not need too much time for the DPM training especially we have the base model and can perform efficient fine-tuning. Also, the case that is discussed (1M images and 120GPU-days for LDM) does not precisely align with the conditional context. These values correspond to the original LDM but we today usually use SD which is trained on huger datasets and takes more time training. Here if we want to say a single condition space tailored DPM may require expensive resources, maybe it is better include cases on finetuning SD or other domain-specific DPM. - For some baseline methos, it is unclear how they are implemented and compared. For example, LDM is selected as the baseline which gives qualitative and quantitative, but I am not sure how it is used and which checkpoint is selected. The authors can clearly indicate how baselines are implemented.

Questions

Considering the overall contribution and the weaknesses, I choose "Borderline reject". I am willing to adjust my score for my potential misunderstanding.

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

The limitations are discussed.

Summary

This paper proposes a new approach to improving the efficiency and applicability of Diffusion Probabilistic Models (DPMs) for various image restoration (IR) tasks. A new modular diffusion probabilistic image restoration (DP-IR) framework combines pre-trained state-of-the-art IR networks with generative DPMs. This framework requires only a small additional module (0.7M parameters) to be trained for specific IR tasks, making it more practical and less computationally expensive. This framework is evaluated on four benchmarks, covering tasks such as burst JDD-SR, dynamic scene deblurring, and super-resolution.

Strengths

Originality: Modular framework and accelerated sampling strategy. Quality: Robust methodology and comprehensive experimental validation. Clarity: Clear presentation. Significance: By reducing the computational requirements and enhancing the generalizability of IR methods, the framework could facilitate the adoption of advanced image restoration techniques in real-world scenarios where computational resources are limited.

Weaknesses

Computational cost analysis: A more detailed analysis of the computational cost, including memory usage and inference time, would be helpful. Experimental validation: While the paper demonstrates strong performance on specific benchmarks, the experiments could benefit from additional datasets to further validate the generalizability of the proposed framework.

Questions

How sensitive is your framework to the choice of hyperparameters?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Elaborate on future work directions that could address these limitations. For instance, proposing specific research avenues to enhance the generalizability of the framework, or developing more adaptive hyperparameter tuning methods.

Summary

This manuscript proposed a modular conditional diffusion model for image reconstruction, consisting of three components: a pre-trained image restoration network, a denoising network, and a fusion network.

Strengths

The model reduces computational load by minimizing the number of network modules that need to be trained and by utilizing lightweight networks. The fusion network is the only component that needs to be trained for each specific IR task. The model also achieves significant acceleration in the sampling process without any loss of reconstruction quality.

Weaknesses

The analysis of trade-offs between model size, speed, and performance may be insufficient. There is insufficient explanation regarding the reasons for selecting specific modules.

Questions

 The standard and rationale for selecting the baseline models have not been detailed. For example, why were specific image restoration networks, denoising networks, and fusion networks chosen? Were these choices based on certain performance metrics, relevant literature, or experimental results? Providing this explanation will help readers understand the rationale behind the model design.  Were other acceleration techniques used during the sampling process? If so, please provide specific technical details and experimental results, comparing performance differences before and after applying these acceleration techniques.  It is suggested that the authors revisit the formula in line 115 to ensure that each part is clear and understandable. If there are any symbols or variables that are not clearly defined, please provide explicit definitions. Consider adding explanatory text before and after the formula to help readers understand its meaning and application.  It is advisable for the authors to verify if the PSNR values in Table 5 are consistent with those in other tables. If all tables should indicate the PSNR Target values as ∞, ensure consistency across all tables to avoid confusion for readers during their review.  It is recommended that the authors redesign the color scheme of Figure 1 to enhance its aesthetic appeal and improve clarity for better understanding.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors do not point out the limitations of the work and do not offer further perspectives. I hope it will be improved.

Summary

This paper proposes a modular diffusion probabilistic IR framework to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs with a light-weight fusion network. Experimental results on burst JDD-SR, dynamic scene deblurring, and super-resolution demonstrate its superior performance and better perceptual quality.

Strengths

1. The proposed framework is based on the existing IR network and equipped with a denoising module, which is suitable for various reconstruction problems without retraining and reduces the burden of computing resources and training data. 2. An accelerated sampling algorithm is proposed to further reduce the computational burden during inference. 3. Experimental results on burst JDD-SR, dynamic scene deblurring, and super-resolution demonstrate its superior performance and better perceptual quality, highlighting DP-IR’s versatility.

Weaknesses

1. The contribution of accelerated sampling seems trivial. As the author said, references [13],[52] already proposed a similar idea. 2. Although the author criticizes the task-specific nature of existing solutions, DP-IR still needs to train the fusion network for each specific task, which makes it less versatile. 3. Absence of necessary ablation experiment: a) The effectiveness of Eq.10’s sampling strategy; b)Inference without image restoration network, i.e., input with y directly.

Questions

1. Why utilize a smaller version of MIRNet as the denoiser, not other architecture used for diffusion? 2. Why not utilize open-sourced diffusion model, like stable diffusion. 3. In Eq. 10, the author gets x_{\tau} through posterior sampling from x_{T} and E[x_{0}|y], why not directly noising E[x_{0}|y] to x_{\tau}? The idea that “Lemma 3.2 makes the final reconstruction quality unaffected” is confusing, because E[x_{0}|y] itself is not the true x_{0}. I think an ablation study is needed to verify your idea. 4. Although the ablation of different combination of image restoration network and denoising network is given, an ablation of only utilize image restoration network or only utilize denoising network is also needed. 5. Minor error: a) “in section 3.4” in line 228 may be “in Table 1”; b) The Table 1 should be table 2?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes.

Rebuttal is online - please respond

Dear Reviewers, Authors carefully prepared their rebuttal - trying to address the concerns you have raised. Please check the rebuttals and join the discussion about the paper. Regards,

Kind reminder of rebuttal acknowledgement

Dear Reviewers, We would like to kindly remind you that the end of the discussion period is approaching soon. We would be grateful if you could acknowledge whether our rebuttal has adequately addressed your comments or if you need any further clarifications or explanations. If you have any additional concerns regarding our work, please feel free to raise them so that we have the opportunity to respond in a timely manner. Sincerely, The Authors