Summary
The paper introduces ImageReward, a new dataset of human preference over generated images given a text prompt. Human preference for images is rated across three dimensionalities: text alignment, image fidelity, and harmlessness. Using the dataset, they train a reward model to score the generated image and text prompt pair. The reward model consists of a small trainable MLP head over BLIP text-image features. The paper further proposes a baseline method of fine-tuning the generative diffusion model using the score model to increase the generated image alignment with human preferences. The fine-tuning loss is a weighted sum of standard diffusion loss and the one from the scoring model (only at the predicted images at lower timesteps of diffusion).
Strengths
The annotated dataset of human preference is one of the first datasets of this kind and scale. This will help in further research in both text-to-image model evaluation and improving the generations with higher human preference.
Both the reward scoring model and fine-tuning method based on the score model are shown to work on par of better than existing baselines.
The paper consists of extensive analysis and details regarding the dataset annotation, scoring model, and its comparison to recent methods.
Weaknesses
1. One limitation of the dataset might be that it becomes less relevant as the generative models improve. Given that the dataset only consists of generated images with their corresponding text prompt and not a plausible ground truth image for the prompt with the best possible human preference score.
2. Only fine-tuning the model on 0-10 timesteps with the scoring model doesn't seem optimal. Specifically, in cases of object omission, the layout has already been decided in the initial stages of diffusion; thus, the reward score guidance at later stages might not be effective. Is there any ablation or analysis regarding what metrics among fidelity and text alignment improve the most?
3. In Eq2, phi is implemented as a ReLU function, as mentioned in line 260. Probably this should be ReLU over the negative of the score function. Because the higher the score, the better. Or is my understanding incorrect?
4. It would be great to expand on the evaluation setup and metrics, which are sometimes not very clear.
(a) In line 122, are the 100 real user test prompts different than the ImageReward dataset used to train the scoring model? Similarly, 466 and 371 prompts in Table 3.
(b) How is the "filter" evaluation metric calculated in Table 3? Does this calculate the number of times the model didn't select the worst image in top-k?
(c) In Table 4, it's unclear how the evaluation numbers are reported. Does it denote the #winrate for each method out of total N samples (N being the sum of the column), or is it a binary comparison of each method vs the baseline? How many generated images per prompt over the 466/77 prompts were used for the evaluation?
(d) Some of the baseline, e.g., reward weighted fine-tuning method, performs worse than the baseline in Table 4. Is there any analysis regarding that?
Questions
Minor point: some grammatical issue in line 290, 302 and 360
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.