LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

Summary

The authors proposed a framework to evaluate text-to-image synthesis with multi-granularity compositionality powered by LLMs. Two big steps are transforming an image into whole-level and object-level visual descriptions in a text to be used for a part of the prompt to LLMs, then LLMs measure the alignment with a detailed instruction prompt to give the LLMScore. They argue that the correlation between LLMScore and human judgments, along with qualitative evaluations with the generated rationale, confirm the superiority compared with commonly used metrics like CLIPScore.

Strengths

They proposed an evaluation framework exploiting the power of LLMs to evaluate the challenging text-to-image generations. Giving the rationale for evaluation justification would be a new feature compared with previous works. The experimental procedure is solid to evaluate the proposed metric, although the comparison is limited to commonly used methods, i.e., cosine similarity or cross-attention using CLIP and BLIP features. The manuscript is well-written, well-organized, and easy to follow.

Weaknesses

W1. The most concern is the comparison with recent work from NeurIPS 2022, "Mutual Information Divergence: A Unified Metric for Multimodal Generative Models" (MID) [17] (the citing number is from the submitted paper). MID similarly exploit pretrained features, i.e., CLIP, but possibly with BLIP, to calculate the text-and-image matching score; however, their method (MID) uses the statistical distribution of the features for finer evaluations than cosine similarity. The authors should compare this recent representative work to show the effectiveness and significance of the proposed method. Specifically, what would be the results of MID using BLIP features for fair comparison? W2. Does visual descriptor have significance? GPT-4 can accept images as inputs, which means instead of visual descriptions (Sec. 3.1), one may exploit the power of LLMs directly, giving a synthesized image as a prompt. Fine-grained examination of visual input would get good results rather than using cumbersome generations of visual descriptions utilizing other pretrained models. This ablation would support the argument for the core role of the proposed visual descriptor. Otherwise, this work would be reduced to a naive application of multimodal LLMs, e.g., GPT-4. W3. Robustness to error accumulations. LLM Evaluator inevitably relies on the accuracy of visual descriptions. Since there is no systemical analysis of the catastrophic degradation from an error in visual descriptions may hinder the reliability of the proposed method. I am eager to reassess my score based on the authors' faithful feedback in handling the issues raised above.

Questions

Figure 2-4 includes the brand logo of OpenAI. According to their brand guideline (https://openai.com/brand#things-to-avoid-when-using-our-logo), the authors should not alter their logo using custom colors. Besides, the authors used the OpenAI logo to represent LLMs; however, it is inappropriate to represent LLMs with the brand logo of a for-profit company in the manuscript.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

L1. This work decisively relies on the performance of pretrained models. 1) image captioning model BLIPv2, 2) GRiT [41] for local region descriptions, and 3) GPT-4 for LLM evaluator. How do you assess the robustness toward potential errors of visual descriptors or LLM evaluators? L2. Social bias and fairness issues can also be inherited from pretrained models. L3. Without the commercial license to use Open API, reproducibility cannot be checked or used. Could you provide the results using the open-sourced LLM evaluators?

Summary

This paper presents a new method LLMScore to evaluate the alignment between a textual prompt and the generated image (especially in the context of the text-to-image diffusion models). The key motivation behind this work is to address the inability of popular vision-language models like CLIP to deal with the fine-grained details in a complex prompt. As a result, in this paper, vision-language models such as BLIP is used to caption the image as well as provide object-centric descriptions. These are concatenated to create a final image description using an LLM. Furthermore, the alignment between the text prompt and the image description is computed using an LLM. Experiments on a variety of text-to-image generation benchmarks indicate that the proposed method LLMScore significantly outperforms prior methods of directly measuring the similarity of image and text embeddings.

Strengths

1) The problem tackled in this paper is an important one, especially due to the inadequacy of existing models. 2) The method proposed in the paper is also quite logical (since using an LLM would be able to perform better fine-grained understanding, as opposed to matching coarse grained features) 3) The experimental results achieved in the paper are quite satisfactory, and show that LLMScore is able to achieve much better results as compared to the standard measures for evaluating image-text alignment.

Weaknesses

One concern about this paper is regarding the following paper [a], which could be treated as contemporary work. While it is understandable that this paper is certainly not required as a comparison as it was released less than 2 months before the submission deadline, there are a lot of similarities between [a] and LLMScore. Both rely on a vision-language model to provide descriptions of the image, and use an LLM to reason over the correctness of the generated image. Of course, there are sufficiently many differences between these 2 works ([a] first generating questions using an LLM and then answering using a VQA model, versus LLMScore generating captions/descriptions first, and then using an LLM to evaluate the description), however, it might be a good idea to atleast acknowledge [a] as a contemporary work in the related work section. Regarding the work itself, an interesting analysis to delve into (given the fine-grained and interpretable nature of LLMScore), would be to break down the errors/inaccuracies into the errors caused due to either the perception models, or with the LLMs (either in constructing the global image description or with the evaluation). Such a breakdown would also enable future work to focus on addressing the weakest aspect of LLMScore to improve it further. Unfortunately, there is no analysis regarding the failure-cases of LLMScore which would have been extremely useful for any reader. [a]: Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, Arxiv 2023

Questions

I have one question regarding the reproducibility of the method. Specifically, with GPT-4, the default temperature of 0.7 is used. In my understanding, this would mean that different runs of the LLM could give different results. Therefore, it might make sense to repeat the experiments multiple times so that the stability of the results are ensured. Of course, if I am missing something, a clarification would be very useful.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, there is a limitations section in the supplementary material, which covers these issues satisfactorily.

Summary

This paper proposes a new automatic evaluation framework named LLMScore with multi-granularity compositionality to evaluate the synthesized image quality of the mainstream text-to-image synthesis models. Its core approach is to tranform images into global and text descriptions through image captioning and region-to-text models, then follow by an LLM to as a visual descriptor and a text-to-image evaluator with rationable output. On six image captioning benchmark datasets, the correlation between its LLMscore and human evaluation results far exceeds the existing commonly used clip-like text-image alignment indicators, reaching into a new SOTA level. Extensive experimental analyzes strongly confirm the usability of the method.

Strengths

1. I truly appreciate the novelty of this paper. It dexterously transform synthesized images into text descriptions, so that it can make full use of LLM's powerful text understanding and reasoning capabilities with a series of hand-crafted constructed Prompts (including sentence analysis, text-image alignment scores estimation, and image generation error statistics). In my standpoint, compared to the previous Clip-like's text-image matching score, this paper undoubtedly set a new T2I quality evaluation paradigm. Aslo, considering the vast influence brought by the AIGC community at present, I believe the impact this paper can bring to the community is significant. 2. The method of this paper is solid, not only in the multi-level extraction of image description, instruction-based and heuristic-based scoring mechanism with rationable generation, but also the considerable performance gains compared with multiple strong baseline methods is also a strong proof. The selected evaluated T2I models are currently the two most popular classes - stable diffusion and DALL·E. In addition, this paper gives ablation experiments of different LLMscore variants and LLMs (GPT3.5 vs GPT4), further confirming the necessity of each component. 3. It brings my great joy is that this work introduces its main ideas and methods, as well as experimental settings and results, etc. in an easy-to-understand manner, so that readers can easily understand.

Weaknesses

Some of the following problems in my perspective may need to solve for the future deployment: 1. The scoring criteria for "Human Annotation" in the appendix should be more specific, and it should be able to guide the annotators to rate the quality of image-text alignment step by step. For example, what is the difference between a score of "4" and "5", is it because of the color generation of the object or the amount is biased? To be honest, if the results of human annotation are not fine-grained and accurate enough, it may seriously destroy people's trust in the usability of the system. 2. The experiment and analysis can be more perfect, but considering that this article is a fundamental work, the time for urgent submission of ddl may not be enough, so this amount of completion is completely acceptable to me. 3. There are several different models applied in the overall evaluation pipeline. If a large model can be unified in the future to complete all evaluation steps (such as GPT4), the evaluation scheme may be more practical.

Questions

What does the average rank correlation of the two items in Figure 5 refer to? Could the author provide a more clearly clarification and revise it in their next edition?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As the author said in the supplementray limitation part, relying on OpenAI's closed-source GPT model and the internal bias of the LLM model are obstacles faced by this proposed evaluation framework.

Summary

This paper proposes a new evaluation metric for text-to-image models. The method works as follows: - Prompt the Blip-2 VLM to generate an overall text description for the image. - Use GRiT to generate coordinates and language descriptions for each object in the scene - Use GPT4 to combine these text descriptions into an overall description of the image. - Prompt GPT4 to score the image, both by asking it for the overall alignment and by asking it to to count the number of errors in the presence/absence or attributes of objects in the scene. They show this score correlates better with human judgment than common baselines.

Strengths

- The approach is significantly more correlated with human judgments than baselines while still being cheaper and faster than human evaluation. - Evaluation is a big problem in text-to-image generation, and better quantitative metrics are useful. - The paper is well-written and clear, and includes a variety of strong baselines. - They break down the accuracy of the different types of evaluation (overall, error counting) so it's easy to see weaknesses.

Weaknesses

The paper has two stages (1) getting a visual description and (2) computing a score. They explore baselines with different variations of (2) but as far as I can tell ablate different versions of Stage 1 (i.e. using just global image descriptors vs just local object descriptors vs both while using the LLMScore scoring method). (I think this would be a useful extension, but the paper is strong without it.)

Questions

Consider mentioning similar concurrent works (e.g. TIFA https://tifa-benchmark.github.io/) - Does the image get penalized for including objects which were not specified in the prompt but make sense in context (e.g. if it asked for a plate with silverware and a cup next to it, does the model get penalized for putting them on a table? Or for including a dining room background which includes miscellaneous objects?) Nit: - Partial phrase on line 218 - Line 217 "default language models" -> "the default language model" - Table 1 caption: "Spearmanr's" -> "Spearman's" - Specify that in the "error counting" metric, lower is better (Fig 7) - Table 3: bold best per column.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The one mentioned limitation is that error counting is still challenging. Not mentioned -- if future works optimize against scores on this metric, models could exploit errors in these metrics. (This is true of all automated metrics on this task though.)

Summary

The authors propose an evaluation metric that focuses on the object-level compositionality and the explainability of the evaluation process. They utilize off-the-shelf models including BLIPv2 (global captioner), GRiT (object detector & captioner) and GPT-4 (large language model). In specific, they first get global caption using BLIPv2, and detect object boxes in an image along with local descriptions with GRiT, and fed the incorporation of these results with an evaluation instruction to GPT-4, which would then produce the score and the rationale according to the instruction. This way, the proposed metric produces assessments that are more correlated with the human judgements.

Strengths

- The proposed metric tackles an important factor in evaluating image generative models. - The authors reveal the potential for utilizing LLMs in evaluating image generative models.

Weaknesses

- The metric would work worse for image domains that are too different from MS-COCO, which is used for training GRiT. (for example, might not work for artistic images…) - The evaluation process consists of many sub-processes, which make it prone to have error accumulation issue. - The metric utilizes GPT-4, which is not easily accessed by many researchers.

Questions

- How many objects are selected from GRiT? Are there any rules? - Isn't there any score variations according to the change in an evaluation instruction? - How much the performance is degraded when the rule-enhanced rating is not used?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

please refer to the Weaknesses section.

Glad to hear the feedback

Weakness 1. Although you could exploit the idea of MID metric to apply without fake image features, it seems out-of-scope and less relevant. As you suggested, the discussion of the difference with this related work would be sufficient. I am glad to hear the feedback and cannot help but raise the score from 5 to 6, leaning toward acceptance.