ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

Summary

- This paper addresses face image restoration using multiple reference images, employing the CacheKV technique to utilize these images effectively. To overcome the issue of loss of appearance information in latent diffusion models, a timestep-scaled identity loss is proposed. - The design choices for CacheKV are explained with experimental results in Section 5.2.1, and extensively discussed in Section 3.2.2. - Additionally, the superiority of the proposed methodology is demonstrated through various experiments using datasets with different levels of degradation, such as FFHQ-Ref-Severe and FFHQ-Ref-Moderate.

Strengths

- The proposed framework effectively utilizes multiple reference images without information loss and provides a suitable framework for leveraging this information within latent diffusion models. - Sufficient experiments validate each proposed method, and both quantitative and qualitative results support the claims.

Weaknesses

- The use of a pre-trained diffusion model redundantly during training and inference results in slower speeds and higher computational costs, posing an obstacle for applications such as video face restoration.

Questions

- How is the time embedding $O$ selected when extracting CacheKV from reference images? - I wonder the results of blind face image restoration of Ref-LDM which get the zero reference image.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Please refer to the weakness section.

Summary

To enhance the performance of blind face image restoration, this paper proposes a reference-based method named CacheKV, which adapts LDM to restore lowquality (LQ) conditioned on several high-quality (HQ) images. As there is no dataset available for reference-based image restoration, therefore, the authors construct a reference-based dataset FFHQ-Ref specifically tailored for this task.

Strengths

-The authors collected a new dataset, which could facilitate related research topics. -The paper proposes CacheKV that effectively leverages the information provided by the reference images, significantly improving the performance of blind face image restoration. - The proposed method performs better than other referencebased methods.

Weaknesses

-The paper does not explicitly explain how CacheKV works. -How does it use varying numbers of reference images when calculating qkv? -How does CacheKV provide a more aligned feature space compared to cross attention? -There are many GAN based blind face image restoration papers (reference based & non-reference based), however, the authors did not compare with them.

Questions

What will happen if we use face image from another identity as refrence image?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

see weakness.

Summary

1. This paper proposed ReF-LDM for reference-based face image restoration with CacheKV to utilize the input reference images. 2. This paper introduced a timestep-scaled identity loss, which considers the characteristics of diffusion models and helps ReF-LDM better learn the discriminating features of human identities. 3. This paper constructed FFHQ-Ref, a dataset comprising 20,406 high-quality face images and their corresponding reference images, to facilitate the advance of reference-based face image restoration.

Strengths

1. This paper proposed ReF-LDM, which incorporates the CacheKV mechanism and the timestep scaled identity loss, to effectively utilize multiple reference images for face restoration. 2. This paper constructed the FFHQ-Ref dataset, which surpasses the existing dataset in both quantity and quality, to facilitate the research in reference-based face restoration. 3. Evaluation results demonstrate that ReF-LDM achieves superior performance in face identity similarity over state-of-the-art methods.

Weaknesses

1. The idea of replacing the key/value in self-attention layers with that of reference images has been widely used in style transfer work. This work follows the same scheme - extracting style features (key/value) from multiple reference images and inserting them during denoising. However, from the results, it seems that the style of reference images affects less on the generation. I wonder the reason behind it. 2. At the first stage of training, i.e., train an LDM with only LD condition, which loss function is adapted here? 3. Since reference images aim to provide ID information during restoration, why not just inject face embeddings of multiple HQ faces as a condition? What are the advantages of the proposed method over the above strategy? 4. For the ablation study of the timestep-dependent identity loss, have the authors tried to apply such a loss after T steps? 5. The proposed dataset does not contain new data, which is basically a re-labeling and cleaning work on FFHQ. Overall, the novelty is not significant.

Questions

Please refer to the weakness section and try to answer these five questions during rebuttal.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations are discussed in this paper. When the face region is occluded by other objects, the proposed model may generate artifacts. For certain face poses (e.g, side face), the reconstructed eyes may appear unnatural.

Summary

This paper investigated the reference-based face restoration model based on latent diffusion model. It proposed a ReF-LDM method involving a CacheKV module and a timestep-scaled identity loss. A FFHQ-Ref dataset consisting of 20,405 HQ face images with corresponding reference images. Experiments are conducted to evaluate it.

Strengths

+ A reference-based face restoration model based on latent diffusion model. + A FFHQ-Ref dataset consisting of 20,405 HQ face images with corresponding reference images. + A CacheKV module and a timestep-scaled identity loss.

Weaknesses

- While the of reference-base face restoration is interesting, from Table 6, the improvement by ReF-LDM is limited in comparison to no-reference based methods. This issue may be alleviated by further improving the fusion mechanism or introducing more plausible metrics. - In the ablation study (Table 3), the effect of timestep-scaled identity loss is given. However, more experiments are required to assess the results by other possible variants of timestep-scaled identity loss. - From the visual results, it seems that ReF-LDM may change the illumination of the degraded images (see Fig.s 8&10.

Questions

One major concern is that the novelty seems to be limited. For example, the authors are suggested to discuss CacheKV with MasaCtrl [1], and discuss timestep-scaled identity loss with the identity loss used in customized generation [2]. [2] is a concurrent work and can also be ignored. [1] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Arxiv 2023. [2] PuLID: Pure and Lightning ID Customization via Contrastive Alignment, Arxiv 2024. For the other questions, please refer to the weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes. The authors have addressed the limitations.

Response to ethics review

> **Q1. No explicit mention of obtaining consent from individuals whose images are used.** All images used in the FFHQ-Ref dataset come from the existing FFHQ dataset. According to the FFHQ dataset, they collected only images with **licenses for free use, redistribution, and adaptation** (for commercial or non-commercial purposes), including licenses such as Creative Commons BY 2.0, Creative Commons BY-NC 2.0, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works. We appreciate your suggestion and will include this information in the revision. > **Q2. Discuss potential biases in their dataset and how these are mitigated.** We will conduct relevant analyses (such as examining race, age, and gender distributions) and discuss the findings in the paper. > **Q3. Outline guidelines for the ethical use of their model and dataset by other researchers or practitioners.** We will add guidelines for the ethical use of our model and dataset in the paper, including: - Cautioning against using the model for malicious purposes such as deepfakes. - Emphasizing the importance of obtaining permission when using reference images. - Recommending transparency when using AI-enhanced images in public contexts. - Releasing the dataset and model under a non-commercial license. Thank you again for helping us improve the ethical aspects of this work.

Response to ethics review

> **Q: The construction process of the FFHQ-ref should discuss how privacy of the data is preserved and potential misuse.** We appreciate the reviewer’s suggestion. We will discuss potential misuse and add guidelines for the ethical use of the dataset in the revision. Regarding data privacy, all images used in the FFHQ-Ref dataset come from the existing FFHQ dataset. According to the FFHQ dataset, they collected only images with **licenses for free use, redistribution, and adaptation** (for commercial or non-commercial purposes), including licenses such as Creative Commons BY 2.0, Creative Commons BY-NC 2.0, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works. Additionally, when releasing the FFHQ-Ref dataset, we will **use anonymous five-digit file names without easily trackable information such as URLs**.