PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference

Summary

This paper tries to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework.

Strengths

1. This paper proposes aligning diffusion models for image inpainting with human preferences by integrating human feedback through reinforcement learning, which improves the quality of generated images. 2. This paper presents a dataset containing 51,000 inpainted images annotated with human preferences, addressing the lack of evaluation datasets for image inpainting. 3. The proposed method can be applied to various applications, such as image extension and novel view synthesis, providing visually impressive results.

Weaknesses

1. There has been a lot of work introducing human preferences into diffusion models, such as Human Preference Score (ICCV 2023), ImageReward (NeurIPS 2023), DPOK (NeurIPS 2023), and D3PO (CVPR 2024). a. This paper does not fully elaborate on the work related to diffusion models with human preferences. b. It is necessary to supplement the comparison with these methods, including differences in methodology and advantages in experimental results. 2. A user study is necessary to evaluate the quality of the generated results.

Questions

See Weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors outlined potential future directions in the conclusion.

Summary

This paper makes the first attempt to align diffusion models for image inpainting with human preferences by integrating human feedback through reinforcement learning. The authors theoretically deduce the accuracy bound of the reward model, modulating the refinement process of the diffusion model to robustly improve both efficacy and efficiency. Additionally, they construct a dataset containing 51,000 inpainted images annotated with human preferences.

Strengths

1. This paper provides theoretical insights into the upper bound of the error of the reward model, ensuring the reliability and accuracy of the reinforcement learning process. 2. This paper presents a dataset for image inpainting tasks that incorporates human aesthetic preferences. This dataset will facilitate further research into the evaluation of image inpainting models and aid in generating high-quality images that better align with human aesthetics.

Weaknesses

1. Since reward models and feedback learning trained based on human preferences have been introduced in Text-to-Image Generation (T2I), it seems they are simply applying the same process from T2I to Image Inpainting. This work should clarify the distinctions from T2I to highlight its innovation. 2. The core contribution of this paper is Equation 11; however, its rationale, implementation details, and effectiveness lack explanation and validation. First, the selection of hyperparameters $k$ and $b$ is not addressed, despite their importance. Second, there is a lack of detailed explanation from Equation 11 to its implementation in the model. Third, the choice of using $e$ instead of other functions needs to be validated. Finally, the description of the ablation study on $k$ in Table 3 is unclear, making it difficult to assess its effectiveness. There is also a lack of ablation analysis for the $b$ and $e$. 3. The weights assigned to the three scores in the manuscript are [0.15, 0.15, 0.7]. Why are these scores combined into a weighted score instead of being used individually? Furthermore, how were these weights determined? The basis for this weighting is unclear. 4. What does the "Rank" metric in Table 2 signify? It lacks explanation. 5. There are details missing regarding the dataset annotation. Such as the number of annotators employed, whether training was required for them, and the amount of time each individual spent on the annotation task.

Questions

Please refer to the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors provide the future work in the last section.

Summary

This paper is the first to use reinforcement learning in diffusion-based image synthesis. This significantly improves the quality since image synthesis is usually a one-to-many mapping, which may not be suitable for conventional learning methods. To generate reward functions for RL, this paper also gathers and releases a new dataset on image synthesis with human preference annotation. Additionally, the paper computes the theoretical upper bound on the error of the reward for more efficient RL training.

Strengths

1. Release a new dataset for image inpainting and outpainting benchmarks with human preference annotation. The generation process of the dataset is well-documented. 2. First incorporate RL with diffusion-based method on image inpainting and outpainting. 3. The authors provide extensive experiments, supplementary materials with results on multiple datasets, and a project page to demonstrate their results.

Weaknesses

1. Minor question on the motivation of amplification factor. 2. Possible missing references.

Questions

1. In L152, how often do these "relatively large errors" occur? Will the relatively smaller errors balance out these large errors? Since the amplification $\gamma$ greatly improves the performance, does this suggest that the errors occur frequently so that the static $\gamma$ do not handle these very well? From Fig.2, it is shown that the range of $\| z \|_{v^{-1}}$ is [0, 1]. Does amplification $\gamma$ for $k=0.05$ and $b=0.7$ have a range of [1.9, 2.0] (which seems narrow and thus static)? 2. How does this paper compare to [1]? [1] leverage the BERT model for reward scoring. How do you justify the choice of human evaluation instead of foundation models for reward function? Do you have ablation experiments to backup the decision? [1] Black, Kevin, et al. "Training diffusion models with reinforcement learning." arXiv preprint arXiv:2305.13301 (2023).

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

N/A

Summary

This paper attempt to align diffusion models for image inpainting with human aesthetic standards through reinforcement learning framework. To train the model, this paper construct a dataset containing 51,000 inpainted images annotated with human preferences. Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, is provided in this paper.

Strengths

1. This paper is well presented and easy to read. 2. This paper provids detailed experiments to validate the effectiveness of the proposed method. 3. This paper have collect a small scale dataset for human preference of image inpainting results, which can be useful for the ressearch field.

Weaknesses

1. This paper seems to use a common reinforcement learning pratice on text-to-image diffusion model. What is the novelty of the proposed method? Why would this practice better than simply finetuning the model on high-quality (human prefered) inpainting data? 2. There are missing citations and comparisons with the following methods: - A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting - BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion - Hd-painter: High-resolution and prompt-faithful text-guided image inpainting with diffusion models 3. Can you provide an ablation of how the training data quantity influence final results? The training dataset only contain 17,000 images and 51,000 inpainted samples, which is quite small in diffusion model training.

Questions

See weekness

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authore have include limitation in the paper, but I would recommend the authore discuss more on the limitation and negative societal impact.

Response to XP8C

## Ethics Reviewer XP8C **Annotators:** Our dataset is annotated by Molardata, a company that specializes in data processing, storage support services, network technology services, and internet data services. The team consists of 25 annotators who have meticulously worked on 51,000 pieces of data. We adhere to local minimum wage laws, offering each annotator a compensation equivalent to 7 US dollars per hour. **Dataset:** We would like to emphasize that all images used in this study are sourced from **publicly available datasets**. Our professional annotators are responsible for scoring the quality of the inpainted images **produced by the diffusion model**. These scores are derived from the degree of **naturalness apparent** in the images, as perceived through human vision. The proposed method is designed to utilize these scores to adjust the model's distribution. Therefore, we want to clarify that no additional information, in terms of ethical considerations, has been introduced. Thanks for the time and effort. We appreciate any further questions and discussions.

Response to reviewer 95Vx

## Ethics Reviewer 95Vx **Potential for Misuse or Dual Use.** The proposed method is designed to utilize human preferences to adjust model distribution. Moreover, **all images utilized are sourced from publicly available datasets**. Thus, there are **no** additional ethical considerations introduced, a point we have also noted to reviewer XP8C. The proposed method is similar to other inpainting algorithms from an ethical perspective. Moreover, to further address the reviewer's concerns, we will also draft a guideline detailing the use of the proposed method. **Social and Psychological Impacts on Individuals Involved in the Dataset.** Our data is **labeled by the professional company**. They have made the data annotation experts **anonymous** in the labeling process, ensuring that **no** personal information can be leaked. Moreover, since there are many workers involved and all labeled data undergoes thorough mixing and shuffling after the labeling process, **it is impossible to identify any personal preferences for a specific individual**. We will add further discussion on this topic in the final version. Thanks for your time and effort. We appreciate any further questions and discussions.