Exploring Diverse In-Context Configurations for Image Captioning

Summary

This paper presents a case study that examines the design of prompts and its effectiveness in generative visual language models for image captioning tasks. The authors delve into four methods for image selection and four methods for assigning captions to create contextually relevant learning samples. To evaluate different prompt design formats, the authors employed OpenFlamingo, an open-source version of the Flamingo paper. Several ablations and experiments were conducted to investigate the optimal inclusion of images and captions in the prompt of the Visual Language Model (VLM) to enhance performance. The authors put forth several findings: 1. Unlike single-modal NLP cases, the performance of the model is significantly influenced by the multi-modal mutual synergy. 2. The authors observe that the descriptiveness and language patterns of the captions have varying impacts on performance. In cases where selected images compensate for descriptiveness issues, better performance can be achieved by using simpler sentence patterns. 3. Additionally, the authors discover that when the in-context images resemble the test image, the VLM may take a shortcut by directly utilizing the in-context captions instead of genuinely learning to generate captions.

Strengths

Generally, this paper is not novel, as there are similar papers in the NLP literature (see [1, 2]). While the authors mention [1] as one of the prior works, they appear to have overlooked [2]. In contrast to the aforementioned prior works, I believe this paper seeks to delve into prior studies in more complex tasks, specifically open-ended generation tasks such as image captioning, and employing visual language models. In terms of quality, the paper explores a distinct aspect of the research questions and presents compelling findings that align with some prior works. Experimentally, the paper demonstrates rigor by thoroughly exploring different prompt constructions. [1]. "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?" EMNLP 2022. [2]. "What Makes Good Examples for Visual In-Context Learning?" presented at ICML 2023.

Weaknesses

The paper has a few notable weaknesses that can be addressed: - The writing, particularly in lines 141-150, is challenging to comprehend. It would be beneficial for the authors to rephrase or provide visual aids to clarify this section, as it contains one of the paper's ablation studies. - Sections 4.2.1 and 4.2.2 should be swapped, considering that in the pipeline, images are typically selected first, followed by the assignment of captions. The current ordering of the text makes it difficult to parse. - It would be helpful to confirm the accuracy of the statement, "Then, as demonstrated in Figure 5(d), even MGC-TF@66 surpasses GTC." - The author's choice of abbreviations throughout the manuscript makes the reading experience challenging. It requires readers to frequently refer back and forth between different parts of the text to fully understand a paragraph. It would be advisable to use less abbreviated phrasing. - Including more elaborate captions for the figures would enhance their comprehensibility as standalone visual aids. - The paper exclusively explores one visual language model, specifically OpenFlamingo, with 9B parameters. It is important to acknowledge that the findings may not be generalizable to other models.

Questions

1. The paper could benefit from considering alternative visual language models to enhance the quality of the results. One such model is FROMAGe [1], which also utilizes a sequence of interleaved images and texts, similar to OpenFlamingo. It would be valuable for the authors to explore the use of FROMAGe in addition to OpenFlamingo and examine if the findings remain consistent across different models. 2. Additionally, it is worth investigating the impact of using smaller language models within OpenFlamingo. As per the OpenFlamingo GitHub repository, there are newer versions pre-trained on smaller LLMs. The authors could conduct ablations with these models to determine how the findings vary when using models with fewer parameters. This would provide insights into the robustness of the results across different model sizes. [1]. Grounding Language Models to Images for Multimodal Inputs and Outputs, ICML 2023

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

There are not any limitations.

Summary

The paper studies how to choose in-context examples (images and text) for open flamingo models on the CoCo captioning task. Extensive experiments are run with many baselines. Based on these experiments, the authors conclude (1) given sufficient descriptiveness, simpler captions are better, (2) similarity of in-context images to the test image is less important when the corresponding captions to said images are poor.

Strengths

S1. Strong empirical investigation of how different image and text sampling strategies affect in-context image captioning performance. The paper is correctly framed and emphasizes experimentation over methodology. S2. Extensive baselines implemented for the investigation. S3. Seemingly strong performance relative to random sampling on the train set, which is traditionally used to generate in-context examples.

Weaknesses

W1. Presentation. Consider adding some of the key findings to the abstract. 7.3 CIDEr points is significant! It may be nice to mention this in the abstract as well. W2. Presentation. Some of the images in Figure 1 are hard to follow, consider making the example more demonstrative, or making the images bigger. Also grouping the good and bad captions within the same bubble was initially confusing and it took me some time to parse what was going on. W3. Presentation. L51-59 potentially have too much detail for the introduction. Consider distilling the content. The key takeaway from this paragraphs seems to be that the authors experimented with many sampling strategies. W4. Experiments. It would be nice to see similar analysis for the VQAv2 dataset, which is also supported in the OpenFlamingo evaluation suite. One natural question is do some of the findings for image captioning translate to VQA-like tasks, which are potentially more similar to the QA tasks shown in Figure 1. W5. Presentation. Consider removing the abbreviations SIIR, DIIR, etc. from the intro. They were a bit tough for me to follow as they are not standard to the best of my knowledge and made reading the intro more difficult. In general, the many abbreviations in the paper make the writing difficult to follow. W6. Related work. Consider adding a related work section on image captioning, since this is a focus of the paper. W7. Clarity. A bit of a nit, but please specify how how the samples are randomly sampled. I am assuming uniformly at random? W8. Clarity. Can you please expand on the SIIR-TAG baseline? In DIIR, how can SIIR-TAG be applied to a collection of images (a cluster)? W9. Missing baseline. What about using models like BLIP2 for synthetic captioning (MGC)? W10. Clairity. It is hard to tell what the takeaways from Figures 3 and 4 are from glancing at them, especially because all of the abbreviations can be confusing. Consider adding they key takeaways to the figure captions. It is not currently clear to me how to read these figures. Is the takeaway that MGC is best? In general, it is not always clear how the conclusions or claims in the paper are supported numerically by the figures or tables (e.g., claims in L231-233). W11. Presentation. In some cases numbers are averaged over multiple shots and in other cases this is not the case. Why did the authors choose to do this? And why is there not consistency in this choice?

Questions

The following questions are distilled from my most significant concerns: Can the authors address the concerns surrounding presentation (W1, W2, W3, W5, W7, W8, W10, W11)? While the baselines seem thorough, they are not always clear for me to follow. Additionally, while the authors run many experiments, the presentation of these experiments is not always interpretable and it is not clear to me that the experiments actually support the conclusions that the authors draw. Some more clarity surrounding this would be greatly appreciated. Is it possible to do some experiments for the best, middle, and worst baselines on VQAv2 to see if trends are similar for another image to text task (W4)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors acknowledge that findings for the open flamingo model tested may not translate to the more powerful, but also closed-source flamingo model.

Summary

This paper examines the effects of varying configurations on Vision-Language (VL) in-context learning, specifically in the field of image captioning. The authors developed four strategies each for image selection and caption assignment to determine how different methods of configuring image-text pairs impact the learning process. The study revealed two insights: 1) Caption quality, influenced by descriptiveness and language patterns, impacts captioning performance, with simpler language yielding better results when paired with descriptive captions. 2) The effectiveness of similar images relies on caption quality, with excessive similarity potentially misleading the model when paired with low-quality captions.

Strengths

+ The general goal is reasonable and interesting. The field of in-context learning in multimodality is well worth exploring. + The experiments were comprehensive. The authors compared numerous in-context learning strategies, including some new proposals of their own. + The paper provides some useful insights about how to choose in-context samples.

Weaknesses

- The authors said that “the performance ... heavily relies on the caption quality” in Fig 1 Caption. I'm curious about how the authors define the quality of a caption. The example given in Figure 1 suggests that the caption in the blue box is better than the one in the red box. It is easy to understand because the red box caption contains clear errors, such as repeated phrases. When selecting in-context samples, such samples obviously should not be selected. But, in practice, such samples should not even be available as candidates, right? Could the authors display what captions the model considers as high-quality or low-quality? Or intuitively illustrate what captions are expected to consider as a high quality among several ground truths? - One insight of the paper is that "When captions adequately describe salient image objects, simpler language patterns may yield better results." But aren't the ground truth captions in the training set in the language style expected by the benchmark? If we use captions generated by the model, then how does the model know what style is required by the benchmark? - I'm not certain whether the experimental setting in this paper can effectively evaluate the capabilities of in-context learning. In-context learning aims to use a small number of samples as demonstrations to learn some new things, such as a new task, new input-output format, new object naming rules, etc. This work conducts experiments on the MSCOCO caption dataset. I'm unsure what the purpose of in-context learning is in this case, as these samples seem quite common for large models. So, what patterns or things are expected to learn by model from these in-context samples?

Questions

My major concerns are about the current setting of in-context learning, and I would like to hear more about how to define the quality of the captions, which is important in choosing samples.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This work investigates the configuration of in-context image-caption pairs for few-shot image captioning in a VL model: OpenFlamingo. Specifically, authors compare four different image selection methods: (1) random selection, (2) image-image similarity based selection (SIIR), (3) image-caption similarity based selection (SICR), and (4) image-image diversity based selection (DIIR). On top of the selected images, the authors also test out four different caption assignment methods: (1) ground-truth caption (GTC), (2) model-generated caption (MGC), (3) Iterative prompting (IP), and (4) Model-generated Captions as Anchors (MGCA). The findings from this work suggest that (1) selecting similar images with the test image is better than random sampling; (2) simpler language patterns in captions may be better if they adequately explain the salient info of the image; and (3)similar in-context images can induce short-cut inference from their captions.

Strengths

Analyses on in-context sampling method for VL models can expand the literature. Especially, the fact that MGCA results lead to better performance than random ground-truth captions is interesting.

Weaknesses

All experiments are conducted on a single open-source VL model; this leads to questions whether these results hold generalizability. Further analyses should have been conducted to backup the authors’ claims: “simpler caption pattern improves caption generation” & “verbalizing the major patterns help identify which GTC provides more detailed info about these patterns”. Please see the question section.

Questions

(1) I find it interesting that GTC performs worse most of the time compared to MGC. Why do you think this the case? I am very suspicious that it is due to the consistent format of captions across in-context samples instead of the simple pattern of the MGCs. However, this is hard to analyze with the current setup. MGCs will have very similar sentence structures for similar images compared to GTCs, which is also shown in Figure 7-(a). I suggest controlling the format consistency across GTCs within the in-context samples. For example, you can set captions to have the same complex structure and see how that results in performance. If simplicity is the true cause of improved performance, setting consistent but complex format should not give improve performance. Also, you can try very diverse format across captions but make them simple in structure. (2) Don’t almost all GTCs include the most salient object in their captions? I find it hard to believe that MGCA improves performance because it finds the GTC that describes the salient pattern with more info. The selected GTCs from MGCA will probably have similar structure with the MGCs, hence it reduces the format variance of GTCs across the in-context samples. What happens if you select the GTC that is least similar with the MGC? (3) When giving the in-context samples, do the images really help in performance? What if you drop all in-context images and give only the captions instead? (4) I think it would be better to swap the location of Figure 3 & 4 with Figure 5 & 6. Is there a reason why you put Figure 3 & 4 first? I found it difficult to go back and forth because the main draft explains Figure 5 & 6 first.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors also acknowledge the limitation that these analyses were conducted on a single model.

Summary

This paper explores the in-context configurations for few-shot ability in Vision-Language Models (VLMs). Therefore, they design four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning. As a result, extensive experiments uncover two valuable insights (1) The captions adequately describe salient image objects and simpler language patterns may yield better results. (2) Excessive similarity might cause VLMs to create a short-cut inference from in-context captions. Finally, they introduce the iterative prompting for images with limited or no ground-truth captions, which boosts the performance of models with an average CIDEr improvement of 7.3.

Strengths

1. **The considered problem is very relevant and timely for the AI community.**. It is urgent to explore the in-context configurations for vision-language pre-trained models. 2. **The observations are valuable.**. They observe that better performance may be achieved with simpler sentence patterns when selected images compensate for descriptiveness issues. Moreover, they observe that the VLM may build a shortcut rather than learn to caption when the in-context images are similar to the test one. 3. **The experiments are comprehensive.**. They devised four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning.

Weaknesses

1. **The selected metric is not convinced.** They use CIDEr to evaluate caption models' performances. However, CIDEr is a statistical indicator related to description length, which is hard to completely reflect the quality of the generated captions. Therefore, it is necessary to explore generative evaluation indicators such as CLIPScore[1] and GPTScore[2] that use pre-trained models. 2. **The choices of VLMs are limited, which may lead to unobjective conclusions.** They just use the OpenFlamingo as the multi-modal learner. The authors should add more VLMs to explore the in-context configurations for image captioning, e.g. MiniGPT4[3], LLaVA[2]. 3. **The observation is a little overclaiming.** The in-context configurations observation from the image caption is not equal to the VL in-context learning. There are many visually-conditioned VL tasks that need to be explored such as VQA[4], and dense image captioning. 4. **The method section of iteratively prompting (IP) is confusing.** It is hard to understand how the algorithm of IP works, and the lack of analysis for IP. [1] Hessel, Jack, et al. "Clipscore: A reference-free evaluation metric for image captioning." arXiv preprint arXiv:2104.08718 (2021). [2] Liu, Haotian, et al. "Visual instruction tuning." arXiv preprint arXiv:2304.08485 (2023). [3] Zhu, Deyao, et al. "Minigpt-4: Enhancing vision-language understanding with advanced large language models." arXiv preprint arXiv:2304.10592 (2023). [4] Tsimpoukelli, Maria, et al. "Multimodal few-shot learning with frozen language models." Advances in Neural Information Processing Systems 34 (2021): 200-212.

Questions

I tend to increase the review score once the following questions are answered well. The following questions are shown in weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Post-rebuttal comments

Thanks authors for their detailed clarifications and for providing the experiment I requested. I have also reviewed the comments made by other reviewers and the authors' responses to them. The authors' rebuttal effectively addressed my concerns, and as a result, I will not be lowering my score.