Can MLLMs Perform Text-to-Image In-Context Learning?

Summary

This paper identifies an important yet underexplored problem - T2I ICL. It introduces a benchmark for evaluating T2I-ICL ability of MLLMs. The authors introduce two approaches to measure the performance of MLLMs. For models with image generation ability, they evaluate the generated image under an I2T retrieval setting. For models that are only able to generate texts, they replace the image content with text descriptions.Finally, the authors identify the challenges of MLLMs performing T2I-ICL and provide two approaches to improve the performance.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The paper is well motivated. It identifies an important yet underexplored problem T2I-ICL. - The settings of all experiments are well controlled and introduced, providing much soundness for this paper. - The analysis in Section 5 seems reasonable and insightful. - The paper is overall well written and easy to follow.

Reasons to reject

- As a benchmark, it would be better to test more models instead of just 6. - The evaluation process is formulated as an Image-to-Text retrieval task via prompts (LLaVA) or embedding similarity (CLIP). But what if the generated image does not belong to any class in the predefined set? In this case, LLaVA may fail to follow instructions and thus fail to provide a valid evaluation result. Hence, it is a major concern that the authors do not discuss such cases. - Texts in Figure 4 are almost unreadable. - The first technique for enhancing T2I-ICL abilities is not surprising. ICL was an emergent ability of (M)LLMs to perform task learning without finetuning them. If we can finetune a model, then ICL would be needless.

Questions to authors

- In Figure 4(c) and 4(d), what is the potential reason of models without image generation capability outperforming the models with such capability? On one hand, “capability tax” would be an interesting point, which means that a model capable of doing both may underperform a model that has a single capability. On the other hand, is it because the label itself is in textual modality, so performing a T2T evaluation would be an advantage for text-output-only models? - To support your argument that “finetuning can enhance T2I-ICL ability”, it’s better to construct a test set which has distinct task types compared to the training set. However, the argument is still not strong and this finding still lacks excitement. - The two challenges are not well solved by the two approaches the authors propose. While solving them is beyond the scope of this paper, it would be more insightful to discuss the potential solutions to the two challenges.

Summary

The paper proposes a new task, T2I-ICL where the goal is to generate an image given a demonstration. More precisely, the authors propose two tasks. In the first, the images define the object (e.g. a car, *object inference*) to generate, and in the second, they define the attribute (e.g. red, *attribute inference*) – and vice-versa for the text. A dataset (CoBSAT) is proposed to evaluate such models, and the authors evaluate different types of models, namely (1) MLLMs than can generate images like Emu/Emu2, SEED-LLaMA, and GILL; (2) LLMs using only the textual description of images. The overall setting is clearly described, and the motivation – to analyze the limits of currents models in this simple setting, is interesting. The evaluation of models is done automatically by using VQA systems, and show that object-inference tasks are more difficult for current state-of-the-art models. The authors analyze why current MLLMs fail, i.e. infering attributes/objects from images and generating images are two difficult tasks and propose ways to improve the current models (by finetuning or using chain-of-thought). Overall, the paper is interesting and well-constructed and demonstrates the limits of current MLLMs in a simple experimental setting.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- a new task with a new dataset - the analysis of why current MLLMs fail

Reasons to reject

- the evaluation protocol is questionable (relying on a VQA system) - the dataset might be too easy in the near future (as demonstrated when finetuning models on the data)

Summary

This paper proposes a new benchmark to evaluate MLLM's capability to perform in-context learning for text-to-image generation tasks，specifically it supports two types of visual inference tasks: object-inference and attribute-inference. The authors collect the images of the dataset from both internet resource and Generative model with a carefully designed object list and attribute lists to examine the MLLM's capability to properly generate the images or image descriptions that align with these proposed lists; Finally, it benchmark both text-to-image generation model and closed source MLLM that can only generate text (e.g GPT4V) and identify some interesting insights: (1) The visual prompt post a significant challenge for in-context learning of the current MLLM; (2) The Image generation task is a much harder task than text-generation even when the required attributes are relatively constrained; The author than proposed two simple solutions to address these two challenges including: chaini-of-thoughts & finetuning, both lead to obvious improvements.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. A new benchmark that aims to evaluate MLLM's capability to perform in-context learning for text-to-image tasks, which was previously under-explored. 2. Through this benchmark, the author identifies two major challenge of the current MLLM: (1) Lack of capability to understand the visual prompt in the in-context learning examples (2)limited capability to generate image compared to the text generation capability. 3. Comprehensively compare multiple models on this new benchmark and identify the challenges the current methods encounter. It also proposes two effective method: fine-tuning and chain-of-thoughts to improve the model's capability of in-context learning.

Reasons to reject

1. Some of the benchmark images come from generative models (e.g., Dalle-E). The qualities of these generated images are questionable. 2. The paper mainly leverage LAVVA to evaluate the performance of the generated image or the generated caption of the image. However, given that LAVVA is a relatively worse model compared to some of the closed model (GPT-4V and Gemini), the evaluation results of LAVVA over these strong MLLM is less reliable.

Summary

Quality: The work presented is of high quality, demonstrating rigorous experimental design and thoughtful analysis of results. The introduction of the CoBSAT benchmark to systematically assess T2I-ICL capabilities across a range of MLLMs is a commendable effort. The methodology section is detailed and includes a clear explanation of the evaluation pipeline, which adds to the reliability of the findings. Clarity: The paper is well-structured and written with a clear narrative that guides the reader through the introduction of the problem, the development of the CoBSAT benchmark, experimentation, and conclusions. Each section builds logically on the last, with sufficient detail provided for replication of the experiments. Figures and examples are used effectively to illustrate key points, enhancing the overall readability and understanding. Originality: This work addresses a gap in the research on multimodal in-context learning by focusing on text-to-image tasks, which is less explored compared to image-to-text configurations. The creation of a dedicated benchmark for T2I-ICL is original and provides valuable resources to the research community. The exploration of fine-tuning and Chain-of-Thought prompting to enhance MLLM performance in T2I-ICL also contributes new insights into effective strategies for multimodal learning. Significance: The significance of this work lies in its potential to advance the field of multimodal learning. By identifying challenges and proposing methods to enhance T2I-ICL, the paper sets the groundwork for further developments in this area. The findings could influence future research directions and have practical implications for applications in product design, personalized content creation, and more. The availability of the CoBSAT dataset and the detailed analysis of MLLM performance also make significant contributions to ongoing discussions and research into multimodal AI systems.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. Significant Contribution to Multimodal Learning: The paper addresses a notably underexplored area in multimodal learning by focusing on Text-to-Image In-Context Learning (T2I-ICL). This study not only extends the current understanding of In-Context Learning (ICL) but also adapts it to a more challenging and less studied multimodal context, which is significant for the field. 2. Introduction of CoBSAT Benchmark: The creation of the CoBSAT benchmark is a major contribution of this work. It systematically assesses the T2I-ICL capabilities of Multimodal Large Language Models (MLLMs) across a variety of tasks. This benchmark will likely become a valuable resource for future research in the area.

Reasons to reject

1. Generalizability and Practical Relevance: The scenarios tested appear highly specific and might not generalize well to other text-to-image contexts. The practical applications described (e.g., product design, personalized content creation) are mentioned only briefly without exploring the operational challenges or effectiveness in real-world scenarios. 2. In summary, while the paper addresses an innovative and relevant topic, these concerns suggest it may not yet meet the high standards required for publication at COLM. Addressing these issues could potentially make the paper suitable for publication. 3. Too much content has been placed in the appendices.

Questions to authors

1. How do you ensure the diversity and representativeness of the CoBSAT dataset? Given the challenges of bias in AI, what measures were taken to ensure that the dataset does not reinforce stereotypical biases, particularly in the context of generating images from text? 2. Beyond the use of CLIP and LLaVA for validating the generated images, what other quantitative metrics were considered or used to assess the performance of the MLLMs on the CoBSAT benchmark? Please provide details on the statistical significance of the results, if applicable.

Thanks

Thanks for your response, I would like to keep the score as it is.

Thanks for your response. We now collected all experiment results.

Thank you for your response. We are here to provide the complete set of experimental results which we could not finish due to the time constraints during the rebuttal period. > Q. Insufficient MLLMs. Here we update the full results of Claude, including 8-shot. The updated figures are uploaded here: [Figure 4(c)](https://hackmd.io/_uploads/S1fF41lSC.png), [Figure 4(d)](https://hackmd.io/_uploads/HyutEyxBR.png). Claude's performance appears comparable to that of GPT-4V, while Gemini continues to outperform all the others. > Q. Experiments on fine-tuning. Below are the experimental results for SEED-LLaMA in both 2-shot and 4-shot scenarios. It is important to note that the the results for each theme are derived from different models. For example, for 2-shot, the results for Style-I and II come from models fine-tuned on all tasks except Style-I and II, while the results for Texture-I and II are from models fine-tuned on all tasks excluding Texture-I and II. | Shot | Fine-tuned SEED-LLaMA | Color-I | Background-I | Style-I | Action-I | Texture-I | Color-II | Background-II | Style-II | Action-II | Texture-II | | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | | 2 | ✗ | .636 | .292 | .088 | .196 | .108 | .360 | **.536** | .164 | **.196** | .080 | | 2 | ✓ | **.752** | **.484** | **.208** | **.272** | **.200** | **.568** | .376 | **.240** | .180 | **.104** | | 4 | ✗ | .612 | .360 | .092 | .044 | .048 | .380 | **.532** | **.140** | **.196** | .148 | | 4 | ✓ | **.748** | **.556** | **.208** | **.256** | **.244** | **.616** | .488 | .112 | .132 | **.216** | After analyzing the complete set of results, our conclusion remains unchanged. The results indicate that fine-tuning on various T2I-ICL tasks generally enhances the models' ability to perform T2I-ICL on unseen tasks. Therefore, for unseen tasks that have not been included in the fine-tuning process, ICL remains necessary for efficiency. --- **Final Note**: Thanks again for providing insightful suggestions. All the discussions above will be included in our final version.