Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

Summary

This paper introduces Pick-a-pic, a large and open dataset of human preference over model-generated images. Using this dataset, a scoring function (PickScore) based on CLIP is trained. The paper finds that PickScore has better correlation with real user preferences compare to existing scoring methods. Furthermore, PickScore can be used as an output selection method to improve the generated image's quality.

Strengths

- The proposed dataset fills an important gap for image generation research. Human preference is an important metric and should be considered when evaluating image generation models. - The dataset is collected from real user, contains a large number examples, and most importantly, is open-sourced. - The learned PickScore shows high correlation with real user preference. Although expected from training on Pick-a-pic, it still provides a valuable alternative over existing evaluation metrics. - It is nice to see the potential of PickScore being used to develop better image generation models.

Weaknesses

- It would be good if the paper could provide a more comprehensive analysis of the prompts used by the users in Pick-a-pic. How diverse are the prompt? What is the word distribution difference between Pick-a-pic and existing datasets such as COCO or DreamBench? - It is possible that the preference of general public users could be biased towards certain aspects of the generated image such as its interestingness, while neglecting other aspects such as faithfulness to the prompt. It would be good if the authors could provide some analysis on the specific types of biases observed in Pick-a-pic.

Questions

- Is it possible to ensemble PickScore with other scoring functions as a more comprehensive evaluation / model selection method?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

It would be good to see a more detailed analysis on the limitations of the dataset and PickScore.

Summary

This paper introduces a dataset of text-to-image prompts, along with user preferences over the generated images. This is done by building a web application where multiple images are generated for a given prompt, and the user is asked to select their preferred image. As a result of collecting this Pick-a-Pic dataset, a CLIP model (with a ViT-H backbone) is finetuned on these user preferences to obtain a scoring function PickScore that demonstrates strong results on evaluating text-to-image alignment. The results indicate that the proposed model PickScore is not only able to significantly outperform standard text-image alignment metrics (i.e CLIPScore), but also performs better than some contemporary work for the same. Additionally, the paper also demonstrates that selecting the best image from a large set of images generated by varying the prompts and seed using PickScore results in images that are preferred over other selection strategies.

Strengths

Overall, I think the paper makes a very solid and timely contribution. 1) Existing methods for evaluating text-to-image alignment have been quite limited, and therefore a large-scale dataset of user preferences that is crowdsourced would be very valuable. 2) Furthermore, the PickScore model can be useful for a variety of downstream applications since it seems to be atleast on par with humans in evaluating text-image alignment. 3) The experiments in the paper are also quite thorough and have comparisons to a lot of different models for evaluating text-to-image, including contemporary work.

Weaknesses

I believe that the paper tackles a problem that is of interest to sufficient audience, and achieves all/most of the claims made in the paper, therefore I do not have major weaknesses to discuss. Naturally, there could be alternative methods/objectives to train better models for improved performance, different vision-language models (i.e BLIP/BLIP-2) could have been considered, or using PickScore to improve text-to-image generation models could all be done in future work.

Questions

My only major question is regarding the "superhuman" performance achieved by the PickScore model. Given that the ground truth is captured from human preferences, I do not fully understand what achieving superhuman performance on this benchmark means (and the description in the paper is quite short lines 176-178). A slightly more detailed explanation of this would be very helpful.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, I think the limitations section in the paper covers the major issues that might arise from this work (i.e dataset having NSFW images, some noise in cases where the users of the applications make mistakes). Apart from this, the paper also mentions that they have not run a hyperparameter sweep, so it is possible for some future work to train an improved model using the Pick-a-Pic dataset.

Summary

The study gathers a dataset of user preferences by soliciting human choices regarding more preferred images in response to text prompts given a text prompt. Besides, the authors propose a metric called PickScore by fine-tuning CLIP-H on this dataset. The experiments show that PickScore demonstrates better alignment with human judgment compared to existing evaluation metrics such as FID or HPS.

Strengths

In this paper, a novel dataset named Pick-a-Pic is constructed, considering human preferences. This dataset proves to be valuable as it can be used to train an automatic evaluation metric, such as the proposed PickScore, to effectively assess the quality of text-to-image models.

Weaknesses

1. It is important to acknowledge that the proposed Pick-a-Pic dataset may exhibit certain biases due to the usage of images generated by existing generative models. As a result, the PickScore metric, trained on this dataset, could potentially inherit the same biases and limitations. 2. Some experimental results are confused, e.g., the correlation value between FID and human judgment is -0.9, which approaches the lower bound of correlation. Does it mean there is a negative correlation between FID and human judgment? Please refer to more details in “Question” below.

Questions

*Dataset and Method The concerns regarding dataset biases and the potential biases associated with PickScore. (1) Considering that PickScore is trained on high-quality images, there is a concern about its ability to accurately evaluate generative models that produce comparatively lower-quality images. (2) Since the images in the Pick-a-Pic dataset are generated by models such as Stable Diffusion 2.1, there is a possibility that PickScore could exhibit bias towards these models, potentially resulting in higher score values. *Experiment (1) The PickScore metric is trained on the proposed Pick-a-Pic dataset, which is annotated by humans. However, it is intriguing to observe that it surpasses human performance in Table 1(b). (2) Confusion about the results of FID. 1. The reason behind the correlation value of -0.9 between FID and human judgment, which nearly reaches the lower bound of correlation, remains unclear. Although the paper claims the reason may lie in “larger scales tend to produce more vivid images (which humans typically prefer), but differ from the distribution of ground truth images in MS-COCO, yielding worse (higher) FID scores”, why the more vivid image would be totally different from the distribution of GT images on MS-COCO? They are even negatively correlated. 2. Which feature extractor is employed in this FID calculation? Is it an ImageNet pre-trained model? If so, is it suitable for capturing the features of the aesthetic or surreal images on the proposed dataset? (3) How about the performance of PickScore on realistic images, it seems most of the images on the proposed dataset belong to aesthetic or surreal ones. *Minor Issues Line 173: Figure 1a -> Table 1a

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This paper mentions the limitations and broader impact in Section 8: “some images and prompts may contain NSFW content that could potentially bias the data and some users may have made judgments without due care”. It would be better to double- or triple-check each sample on the dataset by humans, as the quality of data would greatly affect its usability and reliability.

Summary

This paper contributes a new dataset of human preferences for text-to-image (T2I) generations. It then trains a CLIP-based score function to that can predict human preferences between two generated images. The paper then shows how this can be used to better align a T2I model to human preferences.

Strengths

-- Research angle is very timely on better aligning a T2I model to human preferences. -- The proposed dataset and the learned CLIP-based human image preference function can benefit a diverse cohort of vision researchers. -- The paper is overall easy to read.

Weaknesses

While the proposed dataset itself is a significant enough contribution towards acceptance, the paper's technical contribution is relatively trivial and calls for further discussion: -- the way how the paper leverage a model to learn the human preferences; the paper fine-tunes the entire weights of a pre-trained CLIP and leverages KL loss function to align it to a binary human decision. Given parameter-efficient fine-tuning (PEFT) approach such like LoRA or prompt/adaptor learning has shown exception promise and the fact that the paper only collects a moderate scale of human preferences (less then a million), full fine-tuning might be sub-optimal. The KL loss function choice also needs further ablations, does regression to human preference work? -- On the held-out validation set, the paper's final results are merely around 70% of the time that align with human preference. Given the random chance is $\sim$ 55\%, this is not a particularly good number. The paper needs more extensive discussion to elaborate on this case; is it due to the over noisy human preference ratings that model can not find a globally cohesive way to distil an understanding or the the proposed model is not sophisticated enough to carve out the necessary information for learning? -- The collected human preference ratings are based on synthetic images. What is their impact towards real images? Can the developed human preference scoring function be used as a plug-n-play model that benefits computer vision tasks with need for better human alignment?

Questions

Please see weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A