Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?

Summary

This paper proposes a benchmark for evaluating text-to-image models at commonsense implicature. For instance, given a prompt like "a lightbulb without electricity", an illustration should show a dark lightbulb, not one that's lit up. The paper considers 150 manually selected prompts, grouped in pairs P1 and P2 that cannot cooccur together. Then, models can be scored by how well an image generation for P1 matches P1 and all the things it entails (versus P2). The paper considers humans as well as CLIP and image caption evaluators. Overall, it seems that models struggle at this task, even when provided ultra-descriptive captions. -- overall: seems like all reviews are positive, so should be pretty clearcut to accept this paper!

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

* This seems like an interesting benchmark that could be a useful diagnostic for the community. Many text-to-image model generations are a bit weird and different from how humans perceive the world and create art, and quantifying this gap seems useful. * The experiments seem well done at least to this reviewer. In particular, it seems interesting that ultra-descriptive captions for DALL-E 3 help but do not fully solve the problem.

Reasons to reject

* I am still a fan of this paper, but I'm a bit iffy on some of these prompts in the digital art case. There are ostensibly multiple goals with digital art and representing the physical world accurately is just one of them (along with emphasizing key features). For instance, the melting ice cream sundae example in Figure 5 indeed shows something _in the process of being melted_, emphasizing the second part of the prompt. If there was a way to restrict these models to only generating naturalistic real-world images, maybe this could be avoided, but I think the lack of specification might be a challenge here. I'd guess that more realistic text-to-image models (trained more naturalistic images versus digital art) should do better on this task, which might not be intended. * To this reviewer, it's not 100% clear whether commonsense is a great label for this task as opposed to just simply entailment (like SNLI-style).

Summary

The paper aims to evaluate the capability of text-to-image (T2I) generation models on understanding commonsense. The paper first collects a new manual-curated dataset as a new benchmark for evaluation. The paper also proposes a manual and an automatic pipeline to evaluate T2I model performance on the new dataset. The paper compares a few T2I models including DALL-E, Openjourney v4 and more.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The paper is overall well-written with clear motivation. The paper collects a useful new benchmark for the community to evaluate the capability of T2I models. The paper proposes an automatic pipeline to evaluate T2I capability on understanding commonsense, which is correlated with the manual evaluation process. The paper concludes with a clear direction for future works to improve T2I models.

Reasons to reject

Some details in the methodology are missing. 1. Section 2.2 Page 4, it is unclear how the likelihood score was assigned based on what criteria. 2. Section 2.2 Page 4, how was the data quality control implemented? How to decide data is ambiguous?

Questions to authors

Please see "Reasons to Reject" 1. How were likelihood scores used eventually? I.e. why they were necessary? 2. Page 6 Evaluation Protocol, how was the disagreement between two experts resolved? 3. It will be interesting to know how to enrich/enlarge this benchmark in the future and maintain good quality.

Summary

This paper establishes a novel benchmark, named Commonsense-T2I, for evaluating the ability of text-to-image (T2I) models to understand commonsense in the text prompt. A dataset is created with the help of GPT4 and then manually filtered and annotated. Automatic and manual evaluation metrics are explored to analyze the performance of state-of-the-art (SOTA) T2I models. The paper shows that although SOTA diffusion-based T2I models can generate high-quality images, they fail to properly capture and generate the underlying commonsense expressed by the text input, which indicates a potential direction to improve text-to-image generation.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

- Understanding the commonsense in text input and reflecting it in the generated image is an important capacity of text-to-image generation models. This paper proposes an interesting benchmark to analyze existing T2I models and provides insights for further improving text-to-image generation. - The dataset collection process is clever and reasonable. The powerful GPT4 is used to propose inspirations for commonsense knowledge, followed by manual curation. - The author first conducts manual evaluation and further explores using CLIP or multimodal LLMs for automatic evaluation. The conclusion suggests that multimodal LLMs can make the benchmark's evaluation process much more scalable. - The paper provides detailed error cases of DALL-E 3, indicating that the SOTA T2I models have limitations in 1) understanding the commonsense expressed by the text and 2) correctly reflecting the straightforward "translated/extended commonsense" into pixels. The conclusions of the paper can provide helpful inspiration for future work in the T2I generation.

Reasons to reject

- It is unclear how the categories of commonsense knowledge are determined. If the five categories come from a referred paper, please clarify the source. Also, there is no explanation of how the ratios of the five categories are determined. - The naming and explanation of D1 and D2 are vague and hard to understand -- the name "expected output descriptions" is kind of ambiguous and could mean that the descriptions are the output of some model. There is a typo at the top of Figure 2: The "D1" there should be "D2".

Summary

The paper present a new benchmark called *Commonsense-T2I* challenge. The main contribution is that the author collected a set of text prompts involving commonsense and some of them are long-tail scenarios -- things won't happen often and they won't usually appear in the training corpus. The authors find that most common T2I models fail this challenge significantly. The best among all is DALLE3 and revision of prompts usually won't help -- even though the revision contains the correct commonsense reasonings.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- **Clear motivation**. The motivation of benchmarking T2I model's commonsense is welcome. The benchmark is also very needed given the conceptions that the common mass belief T2I is a solved problem but in fact it's not. With the introduction of commonsense T2I, people would see a new direction of improving T2I models. - **Complete Ablation Study**. Though ablation study for datasets/benchmark papers are not fully required but it's nice to see the authors provide the ablation of different evaluator (automatic ones such as GPT-4 and Gemini, together with human raters). It's nice to see their consistencies and would confirm the benchmark is well designed.

Reasons to reject

1. I would recommend the authors expand the number of T2I models used for evaluation. Probably including recent latent consistency models and SD-3 to see if they have different biases against SD-21 and SD-XL. 2. I would love to hear more about the authors comments on potential ways to improve commonsense T2I. As the authors can collect these amount of benchmark data, the data can also be used to fine-tuning T2I models. For example, using fully-automated methods such as DreamSync and its data-efficient followup work SELMA, improving commonsense T2I seems possible and these methods are very simple baselines. I would love to see the authors reasonings on why these methods can or cannot improve commonsense T2I. If they can't, it could further strengthen the necessity of having this benchmark.