Summary
This paper makes a twofold contribution: namely introducing a new metric for evaluating text-to-image alignment, as well as a method that builds upon this insight to improve the process of text-to-image generation.
For the first contribution of evaluating text-to-image alignment, the paper employs the strategy of using a Large Language Model (LLM) to decompose the textual prompt into multiple components, and then using a Visual Question Answering (VQA) model to answer the individual components/assertions to obtain a final score for the alignment between the generated image and the textual prompt.
To improve the image generation process, the paper builds upon the existing approach Attend-and-Excite, that uses the cross-attention maps to focus on different tokens in the textual prompt. Here, the weights applied to the sub-prompts, as well as the strength of the cross-attention are used as parameters to be optimized in an iterative refinement procedure (while optimizing the decompositional metric) to obtain more faithful generated images.
Strengths
1) The paper tackles an important problem (of both evaluating text-image alignment, and improving text-to-image generation), and proposed interesting ideas for the same.
2) The Prompt Decompositional Model intuitively makes a lot of sense, and quantitative results indicate that it seems to be significantly better than prior methods (i.e CLIPScore)
3) Using the improved evaluation metric to improve the generation is a good idea. While in terms of novelty, it is almost entirely based off Attend-and-Excite and prior work that uses prompt weighting, optimizing it during test-time is new, and seems to bring good improvements.
4) The experiments seems to be reasonably thorough. There are user studies that validate the key claims (of improved text-image alignment evaluation, and improved generation), as well as a new dataset of prompts.
Weaknesses
One major concern I have with respect to the evaluation metric proposed in the paper is its similarity to [a]. While [a] is only an Arxiv preprint, and can definitely be considered as contemporary work, I would think that it is a good idea to atleast acknowledge it, since the idea behind breaking down a prompt into multiple Question-Answer pairs and then using a VQA model to evaluate it has been first introduced there.
In terms of the quantitative results, my worries are that the paper mostly relies on user studies/human evaluations to compare the proposed method against existing methods. While this is understandable (and quantitative results on the DAScore metric would be unfair, since the model is explicitly optimizing it), it might make sense to have quantitative results on other metrics (such as T2TScore) to demonstrate that the improvements are visible on automatic quantitative metrics. This would also allow benchmarking on other datasets beyond the Decomposable Prompts.
[a] Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Questions
The only question I have is that all the experiments in this work are done on the Decomposable Prompts dataset. While it's large enough (with 4k prompts), I wonder if it is sufficient in terms of variety, of if it makes sense to also benchmark on other datasets such as DrawBench.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Yes, the limitations section in the supplementary material provides a fairly comprehensive overview of the possible limitations of the proposed method in the paper.