PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance

Summary

The current restoration approaches based on diffusion prior rely on prior knowledge of the degradation process, and thus fail to seamlessly adapt to different scenarios. Motivated by this, the paper proposed a " partial guidance" approach to directly modeling the distribution properties of high-quality images and then exploit it to serve as guidance for the diffusion process. The extensive experiments demonstrate the advantage of the proposed approach.

Strengths

- The proposed method diverges from traditional practices of modeling the degradation process, and instead, focuses on modeling the desired visual properties of high-quality images. The learned vision cues are then served as diffusion guidance for the generation process. This motivation and the proposed model are technically sound to me. As demonstrated in the paper, it showcases adaptability to different degradation situations, yielding outstanding results. - The experiments conducted in the study seems solid, featuring comprehensive comparisons with various baseline approaches.

Weaknesses

- For face restoration, it's important to make sure the restoration is performed in an identity-preserving way, and quantitative evaluation regarding this is crucial. However, such quantitative evaluation seems to be missing in the paper. - To clarify further, I have no doubt the image quality of the proposed approach outperforms the baseline, but I'm concerned if the quality is improved at the cost of degradation in identity. So it's important to get a quantitative evaluation regarding this

Questions

How would the proposed method compare to the pre-trained text2img diffusion model? - Let me explain for further clarification: with the current pre-trained text2img diffusion model (stable diffusion, for example), one can perform image enhancement using img2img pipeline, where you start with an inversed image (plus a small amount of noise) and guide the diffusion process with text prompt like "a high-resolution high-quality photo". It would be interesting to add such an experiment for comparison. - for you reference, by "img2img" pipeline I mean: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No major limitation in terms of societal impact

Summary

This paper proposes to use some simple properties to guide the reverse diffusion process. The proposed approach makes no assumptions about the degradation process. This paper also shows many different face restoration visual results to demonstrate the superiority of the proposed method.

Strengths

1. The proposed method has been experimented with in many different face restoration tasks, which demonstrates that the proposed method can be easily adapted to other face restoration tasks. 2. Overall, this paper is well-writing and easy to understand.

Weaknesses

1. The main contribution of this paper is using some simple properties to guide the reverse diffusion process, which is inherited from classifier guidance [7]. What is the technical and original contribution of this paper? Please discuss the technical differences between the proposed work and classifier guidance [7]. 2. How to choose the properties that are used in different tasks? This deserves an ablation study. Please show the influence of some different properties. 3. This paper claims one of the advantages of the proposed method is no assumption of the degradation process. As far as I know, the DifFace also does not need to know the degradation process. What is the advantage of the proposed method against DifFace? 4. No quantitative results are found in the manuscript. Only visual results are not enough to evaluate the proposed method. Please provide a fair quantitative comparison against other comparing methods such as commonly used datasets CelebA-Test, and three real-world testing datasets proposed in GFPGAN. 5. The inference time is also important to evaluate the proposed method. Please provide the inference time comparison again with other comparing methods. 6. Some failure cases can contribute to understanding the limitation of the proposed method. Please show some failure cases.

Questions

Please see Weakness.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This paper proposes to use some simple properties to guide the reverse diffusion process. But this paper has not discussed the limitations of the proposed method. I suggest the author discuss the potential and limitations of the proposed method in more detail such as showing some failure cases, which will make the contributions of the proposed method more significant.

Summary

This paper proposes partial guidance, an approach exploiting pre-trained diffusion models for face restoration. Instead of making assumption about the specific degradation process, partial guidance models properties of high-quality images such as structure and color statistics to implement classifier-guidance during the reverse diffusion process. This approach suits a range of restoration task and can be extended to composite tasks. Experiments demonstrate the effectiveness of the proposed method.

Strengths

- This paper considers exploiting image properties irrelevant to specific degradation process to tackle versatile face restoration problems, which has demonstrated effectiveness in experiments. - The qualitative results of the proposed method provided in the paper and the supplementary material are sufficient and impressive.

Weaknesses

- Except blind face restoration task, more quantitative results are expected in the remaining tasks like face colorization and face inpainting. - In blind face restoration task, the pretrained restorer (such as Real-ESRGAN stated in the supplementary material) is employed and finetuned to predict smooth semantics, but these intermediate results are missing, and the improvement of the proposed method compared to the pretrained/finetuned restorer is not demonstrated or analyzed.

Questions

- In blind face restoration task, in addition to real-world data, why synthetic data (like degraded CelebA/CelebA-HQ) is not used for evaluation of more quantitative metrics (like PSNR, SSIM, LPIPS), which is adopted in previous work like DifFace. - Other concerns have already been mentioned in Weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the authors have addressed the limitations and potential negative societal impact of their work.

Summary

This paper proposes a novel solution for blind face restoration. Instead of modeling the degradation process, the authors propose to model the desired properties of high-quality images as classifiers. Similar to guided diffusion, the authors guide the diffusion generation process with specific classifiers to achieve image restoration. Visual results well proved the effectiveness of the proposed methods.

Strengths

1. The idea of modeling the desired properties of high-quality images as classifiers is novel and interesting. 2. The proposed method can solve blind image restoration and can restore images following the reference properties, e.g., color and identity, which is novel and practical. 3. The paper is well-written and easy to follow.

Weaknesses

1. There seems no quantitative evaluation. It is not persuasive enough with only visual comparisons. 2. The setting of hyperparameters may be difficult. 3. Some overclaims. For example, the author said in line 115, "Our partial guidance does not assume any prior knowledge of the degradation process." However, PG needs to know the degradation type. For inpainting, the mask is also needed. (2)

Questions

1. Please see the weaknesses. 2. I am curious about the effectiveness of PG on natural image restoration. For example, what about applying PG on stable diffusion? 3. For the colorization task, the lightness constraint is linear and can be analytically solved in a similar way to DDNM. I wonder if it works to simply apply Color Statistics to DDNM?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Though the evaluation is poor, I still give a relatively positive score for the time being and hope that the authors can supplement quantitative experiments (and subjective questionnaires for blind restoration) and objectively discuss the problems existing in practical applications. After all, this work is about blind restoration. If there is a new method that works well, it will be very helpful to the community.

Official Comment by Reviewer hvsJ

Thanks for the rebuttal and only some of my concerns are addressed. Given the extensive nature of the revisions required for the manuscript, I maintain my recommendation.