What You See is What You Read? Improving Text-Image Alignment Evaluation

Summary

This paper aims to develop a method for evaluating the level of semantic alignment between text and image. In order to achieve this, the authors construct datasets of image and text pairs, and collect human judgments to determine if the text and image are semantically aligned. The paper proposes two methods for estimating alignment between text and image. The first method generates questions and answers from text and checks whether a Visual Question Answering (VQA) model provides consistent answers to the questions. The second method is an end-to-end classifier that predicts if the image entails the text. Experimental results demonstrate that the proposed methods outperform existing multimodal models.

Strengths

- A significant contribution of this paper is the collection of human judgments on text and image alignment. This will aid in the development and validation of alignment evaluation methods in future research. - The annotation pipeline is carefully designed, and the details are throughly described.

Weaknesses

- The VQ^2 method depends on the sampling of questions and answers, which may result in output variation due to chance. The effects of sampling are not discussed in the paper. - Language models used to generate contradicting captions may introduce biases in the datasets. For example, generated captions might have typical wording patterns that are difficult for humans to detect. Validating synthesized datasets is challenging.

Questions

- Can the authors provide more details about the PaLM instruction used to generate contradicting captions? Was the prompt in Figure 2 utilized to rewrite captions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses two limitations: 1. The difficulty in judging alignment between a text and an image even for humans. 2. The challenge of filtering offensive content from the dataset.

Summary

This paper addresses the problem of text-image alignment evaluation by proposing a benchmark named SeeTRUE and two alignment metrics named VQ^2 and end-to-end VNLI. SeeTRUE covers a variety of real and synthetic images and captions. The synthetic captions, which are generated by large language models (LLMs), preserve most words from the original captions but convey a contradictory meaning. The synthetic images are generated by text-to-image generative models. The real image and captions are collected from existing datasets. In addition to the image-text pairs, SeeTRUE also provides binary human annotation of the alignment between these pairs. VQ^2 first generates question-answer pairs based on the caption and then assesses question-answer pair alignment against the image. End-to-end VNLI directly fine-tunes a visual NLI model to determine whether an image entails a caption. Experiments show that the proposed image-text alignment metrics outperform state-of-the-art (SOTA) vision-language models on the SeeTRUE benchmark. The VQ^2 metric also exhibits (1) a high correlation with humans in the evaluation of text-to-image generative models and (2) strong potential as a reranking technique to select generated images.

Strengths

* The proposed SeeTRUE overcomes two limitations found in existing benchmarks: a primary emphasis on real images and a lack of challenging negative captions. SeeTRUE provides a valuable testbed for evaluating image-text alignment methods. * The proposed two image-text alignment metrics are well-designed and exhibit very promising results, which can facilitate the evaluation and development of image captioning and text-to-image generation models. * The experiments are comprehensive and well-designed. * The paper is well-written and easy to follow.

Weaknesses

The current manuscript still has three problems (which do not warrant rejection): * SeeTRUE primarily focuses on the challenging scenario where negative captions differ from positive captions by only a few words. However, it remains unclear whether the proposed VQ^2 and End-to-end VNLI models, specifically tailored for this setting, can still surpass the strong baselines in the standard scenario where negative captions describe completely different images. * Considering the utilization of three models—namely, an answer generation model (T5-XXL), a QA model (T5-XXL) and a VQA model (PaLI-17B)—VQ^2 might incur higher computational costs compared to the baseline methods. It would be beneficial to provide a detailed account of the computational expenses of these methods in Table 2. * It seems that the results in Table 7 of Appendix A3 are missing.

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the “Weakness” section.

Summary

The authors propose a benchmark and two methods for evaluating fine-grained and complex image-text alignment. Their benchmark involves multiple distracter captions and images involving both real and synthetic images and captions. They propose two methods - one evaluates image-text alignment by asking multiple entailed questions, and another is a large Vision-Language model fine-tuned on their benchmark image-caption data.

Strengths

Evaluating by visual entailment is neat since it is interpretable. They also show that they outperform a contemporary benchmark TIFA that does the same. They release a comprehensive benchmark, a combination of available data and annotations they collect, which will be valuable to the community for evaluating vision-language models.

Weaknesses

Sometimes certain design choices are not well motivated and make the reader wonder why such a choice was adopted. For example, when generating qa pairs, the question is answered using a t5 Language model first to filter some qa pairs. Is this to ensure the answer semantically makes sense for such a question in general? If so, writing out clear motivation before explaining what is being done might be good. The benchmark seems to be a collection of simply more human-annotated image-text data. Motivating why sometimes synthetic images or text is required and what are new ways one can use this to evaluate their models (that current benchmarks lack) would be nice. For instance, with the contrastive captions, one can evaluate compositionality. However, we can already do this with Winoground; what extra does this benchmark give us? Some other relevant image-text benchmarks that evaluate compositionality using similar ideas of distractor captions are CREPE (https://arxiv.org/abs/2212.07796) and COLA (https://arxiv.org/abs/2305.03689). It might be good to add in a discussion of how this benchmark relates. (minor) Even though synthetic images and texts are human evaluated and filtered, the datasets may be biased to having images that current generation models can already generate well, limiting the application of evaluating image generation models on the dataset on edge case prompts.

Questions

What are the train, val, and test splits of SeeTRUE? If I understand correctly, the VQ2 models are not trained and are just evaluated on the test split. However, the VNLI is trained on the train splits of SeeTRUE and then evaluated on the test splits. Is that correct? How many questions are asked for a given image-text pair in the VQ2 method?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors discuss limitations sufficiently

Summary

This paper introduces SeeTRUE, a benchmark for evaluating image-text alignment, encompassing a diverse range of both real and synthetic images and text. The authors proposes two innovative approaches to evaluating alignment: VQ2, which relies on question generation and visual question resolution, and VNLI, which relies on fine-tuning substantial multimodal language models. The suggested methods perform better on a variety of alignment tests than earlier methods, particularly in difficult scenarios involving complicated composition or strange images. The study also shows how these techniques may rerank generated image candidates and pinpoint particular misalignments. This provides alignment evaluation methods for image-to-text and text-to-image models.

Strengths

- This paper introduces comprehensive benchmark, SeeTRUE covers real and synthetic text and image pairs from a variety of different tasks, allowing for a more thorough evaluation of text-image alignment models. This could be a solid contribution on this field. - The paper presents two new text-image alignment evaluation techniques, VQ2 and VNLI. these methods outperform previous ones, especially in complex or unnatural images. - The paper is clearly written, and easy to follow - The author shared part of their codes

Weaknesses

- Because performance is dependent on the LLM, the comparison doesn't seem fair. - Also, as Generating Contradicting Captions (even with LLM) isn't a new concept, there isn't much insight to be gained from this paper. ( for example, Momeni et al. https://arxiv.org/abs/2304.06708 ) - There aren't many examples. Even the appendix is not sufficient. It would be better if there were more examples.

Questions

-Regarding the Dependency on the LLM, the methods largely depends on the quality of the LLM and language-and-vision model. Does this mean that the proposed methods would perform poorly if the model is not good enough? Can you provide a comparison of how these methods perform with different LLM and VLM models? -On the Novelty of Generating Contradicting Captions, Since the generation of contradicting captions is not a new concept, how does the proposed negative mining with LLM methods in this field? Is there an additional aspect of the method that brings unique contributions to the field of text-image alignment? -Can these methods be adapted for other tasks that involve the interaction of text and image

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The author addressed the limitations of this work. The authors could discuss the limitation of prompting LLM in more detail, as well as suggest ways to mitigate it.

Post rebuttal response

Thank you for your efforts in answering my questions. After reading the authors' rebuttal, I have increased my rating to a 6 - Weak Accept

Thanks for your response

Thank you for your responses. The rebuttal provides clarification to my questions. I am increasing my score.

Response to Author Rebuttal

Thanks for your response. My first concern is well addressed. As for the second concern, I agree that sequential pipeline is advantageous in the three aspects. However, I think it would be better to report the computational cost (e.g., time cost and the amount of GPU memory used) in the main results of Table 2.