Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models

Summary

This paper studies the problem of getting models like CLIP to perform compositional reasoning. One observed problem with these models is that they devolve into "bags of objects" models, and this work seeks to address their variable-binding ability towards identifying more complex relationships. This paper takes a data-first approach, hypothesizing that the data behind these models might be flawed. The paper starts with the Conceptual Captions 3M dataset. The quality of captions is improved by: 1. Using BLIP2 + OPT to recaption the images 2. Using a GPT-Neo-2.7B model to perform "knowledge expansion". This hallucinates extra text for the given caption, though it should be noted that this is addressed by MIL losses that are introduced. 3. Using segment-everything to get a bunch of regions, which are then fed to BLIP@. 4. Using "negative text augmentation" to change words around to form captions that don't match the image. The paper studies CLIP models that are finetuned on CC3M, using LoRA to reduce catastrophic forgetting. The CLIP losses are extended to incorporate the negative text augmentation strategy as well as to address the knowledge-expansion hallucination issue with MIL. The paper evaluates on VL-Checklist, ARO, and Elevater, which are all datasets for compositional reasoning. Linear probing results show that the model performs better on these datasets with benefits on compositionality. --- update: increasing my score from 6->7 as my main concerns were resolved. I vote to accept this paper. I think I'm in agreement with the other reviewers, except for bpWN whose concerns I'm not really understand (they feel like curious questions IMO but not reason to reject the paper).

Strengths

Overall I am a fan of this paper. I think it addresses an important question: how to make CLIP models more robust to compositional reasoning and variable binding, and it does so by addressing an often-understudied aspect, the role of data. The experimental results and ablation study seem solid to this reviewer, and suggest that the various recaptioning / data augmentation strategies help.

Weaknesses

To this reviewer, the paper seems strong overall. However, I think the collection of caption augmentations seem a bit complicated and so I'm left wanting to know which is the most effective in terms of overall "bang for your buck". For Figure 3 I'm not convinced by the use of CLIP score here. If the paper presents ways of exceeding CLIP performance on compositional data then I'm not sure if CLIPScore is a reasonable metric for evaluating captions that are truly better than CLIP. I would also prefer if these improvements were tried on a larger CLIP model, as ViT-B/32 seems rather small. See the questions below - I think also they could improve the paper if addressed.

Questions

* What strategies are the most effective in terms of pretraining/inference compute? * Do these results hold across model sizes (e.g. CLIP models and recaptioning models)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

thanks for adding a limitations section!

Summary

The current vision-language (VL) models suffer from "object bias" issues, this paper proposes two components to improve the compositional reasoning capability of the existing VL models. It employs the pretrained LLMs to augment the captions of the images, in terms of quality and density, then finetune the CLIP models on CC3M corpus, and demonstrates promising results on two benchmarks.

Strengths

1. This paper employs the strong pretrained LLM to enhance the caption quality and quantity of the images, and finetunes the pretrained CLIP on the augmented CC3M corpus, and shows the promising results on two compositional reasoning benchmarks.

Weaknesses

1. An incremental work to improve the compositional capability of the VL models by augmenting text captions with images, and show decent performance on the two benchmark, and weak performance on the Elevater benchmark.

Questions

1. I assume the the first row in the Table 4 is the CLIP finetuned on CC3M baseline, right? If yes, it looks hard negative captions are key component to improve the performance (row 3 in A), and quality looks not an important component comparing row 0 in B and row CLIP, it gets worse.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper highlights two problems with existing image-text datasets that make them unsuitable for use as pre-training datasets for evaluating performance on attributes and relations, and claims that the models simply act as bag of words when trained on these datasets. The first problem they identify is that often in these web-scraped datasets, the text is not actually describing the contents of the image, but instead is describing how the captioner feels about the image or unrelated information entirely. The second is that the captions often only describe a part of the image. They propose two approaches - captioning images to improve quality and density as well as a fine-tuning approach that is able to handle the noisy captions generated by their method. Edit: I have updated my score to reflect the rebuttal

Strengths

## Originality and Significance * The paper attempts to address two important issues with pre-training on image text datasets (quality and density), which could improve the performance of vision and language models * They use two interesting approaches to improve density of captions - using an LLM to probe for other things that can be said about a given caption (without access to an image) and based on SAM. * They evaluate performance on image classification benchmarks to see performance compared to the base model (CLIP)

Weaknesses

## Major issues * "the CLIP matching score statistics are significantly improved by replacing the original captions with BLIP captions (Fig. 3a), and qualitatively they are a better fit in many cases " The conclusion that BLIP2 based captions are better than the original captions is drawn by computing the CLIP matching score and observing that these are scores higher, which is expected since the BLIP2 model is based on CLIP and would understandably score these outputs higher. This is insufficient evidence and needs to be supported either by human judgement or other approaches (it is unclear what is meant by qualitatively they are a better fit in many cases <-- was a study conducted to conclude this?) * LLM expander approach : "Indeed, the source caption is the only information provided to the LLM, and some facts are likely to be hallucinated and unrelated to the corresponding image." <-- is there some statistic on how often this happens? Some human analysis on a random selection of images that verifies how many expansions have hallucinated facts about the image would be useful to judge the usefulness of this approach. * The SAM expander section does not provide any information on how captions are created from the segments. How are they provided as input to the captioning model? Is it just a crop of the single segment? If it is multiple segments how are they handled? * It is unclear to me how performance on relations (Fig 3) can improve using the SAM approach when as far as I understand, it is only possible to obtain single object using the pseudolabelling approach described in the paper. * It seems odd to me that BLIP-2 does so poorly on the ARO benchmark given that it is a generative model (13% on COCO and FLICKR when CLIP gets more than 3x). Especially when Table 4 Row B(first sub row) shows performance on ARO using captions generated by BLIP and it is around 29 & 40%, and the model on COCO-captions achieves more than 140 CIDEr. Could the authors please describe how BLIP2 was evaluated? (See Table 6 in [1] which shows that even a blind LM decoder can get 99% on the COCO and FLICKR splits that test for word order). My guess is that both ARO and this work use the ITM head - which would not be as good as scoring the likelihood of the caption under the LM head. * Using BLIP-2 to caption and then using those captions as data is the same as distilling information from the dataset BLIP-2 was trained on. An experiment that is missing, would be to compare a model trained on BLIP-2 training data (having CC12M, COCO, VG, SBU) with regular fine-tuning vs the approaches described in this paper as the improvements could be due to access to this larger source of image-text data. ## Minor issues * Fig 3 (a) is unclear - what are the X-axis and Y-axis depicting? Also (c-e) * Negative example generation is not sufficiently described in the paper [1] Image Captioners Are Scalable Vision Learners Too. Michael Tschannen et al 2023

Questions

In Table 4 Row B : Could you explain why adding the LLM expansion would increase the score on COCO and FLICKR ordering subsets of ARO by 2x (row 1&2)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The authors propose a method for data augmentation on image-text web corpora they call DAC. The main idea is to run both a segmentation model and an image captioning model, and then use those outputs to generate captions that are likely to describe the image using an LLM. Finally, a MIL loss + LoRA over CLIP is used to align the image with the autogenerated captions, and anti-align with some machine-generated false negatives. The authors achieve strong performance gains over vanilla CLIP (and no performance degradation in usual vision tasks) using this method when training on the CC3M dataset. Somewhat surprisingly, they actually don't even use the image captions in CC3M (L261).

Strengths

- The topic of making CLIP-style models better compositional reasoners is interesting. - The proposed approach is straightforward, creative, and promising. I like that the authors used LoRA to maintain the original strong performance of CLIP (and specifically test for that). The results are strong compared to the CLIP ViT-B/32 baseline. - I admire that the authors used a public source LLM instead of OpenAI! - Ablations validate that all the pieces of the proposed pipeline play a role.

Weaknesses

- I am not the biggest fan of the framing that web captions are "low quality." After all, these captions were presumably written by humans for a purpose other than ML training --- so they may be low quality for that purpose, but L43-44's blanket description (and elsewhere in the paper) I thought could be made more precise like in L160. - It would have been nice for the authors to front the fact that they actually just use unlabelled images earlier --- it's neat. - If SAM outputs are handed to BLIP in a single-segment fashion, how are the relations in figure 2 generated? e.g., "Miniature zoo of toy animals on a mat" seems to be something that would require multiple segments to parse, and I don't see how this happens given the description in L179. - It would have been nice to see scaling plots --- do these same results apply to larger versions of CLIP? And, do these results cleanly scale with the number of unlabelled images?

Questions

Overall, I liked this work! It gives a nice, data-bottlenecked/auditable method for transferring the compositional knowledge that /feels/ like it should be in/derivable from other models (like SAM) into CLIP, while maintaining the positives of CLIP. My biggest gripe was that the authors didn't apply the method to larger CLIP models, or give scaling results to show how this method might scale with (readily available) unsupervised image data. UPDATE: The authors have addressed my concerns and I have raised my score accordingly.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The authors briefly discussed limitations in a vague sense. But could more be said about the potential risks, e.g., of propagating errors that BLIP2 makes to downstream models? Does DAC result in augmented data with more (social?) biases than the original human-authored data?

Summary

The paper proposes methods to improve performance on compositional reasoning tasks — by improving the caption quality (alignment) and density (description bias). The paper observes that the poor compositional task performance is due to these limitations in pretraining / fine-tuning data. The approach itself, is quite simple — data augmentation using existing Vision, and Vision-Language models. The noisy augmented data is utilised for fine-tuning with an appropriate multiple-instance loss function, which results in significant improvement in downstream compositional reasoning tasks across datasets.

Strengths

(1) Paper is well-written. The approach is described clearly, and obvious issues (e.g., the hallucination introduced due to LLM-based knowledge expansion) are analyzed and discussed. (2) It is clear when visualising the noisy web-scraped training data for VLM models that the captions often are not factual descriptions of the contents or activity of the scene. So, many of these samples are unsuitable for learning a good vision-text representation that can be used for reasoning about object-relationships and other tasks. This was explored in detail [76]. The proposed solution — *generating* captions that are better aligned, and fine-tuning the model on that, is simple but somewhat surprisingly, is sufficient to improve downstream reasoning task performance. The paper also validates the generated captions with CLIP matching score. (3) Increasing the density of captions for a given image by leveraging recent improvements in the segmentation task, is a very reasonable direction to overcome people’s bias to describe only certain “interesting” elements of the image. This aspect has been discussed in the past in the image captioning literature. The paper proposes noisy augmentation techniques, that improve the learned representation for downstream tasks. They key is training with MIL which seems to account for the noise introduced into the training samples. (4) There is significant performance improvements in the reasoning tasks.

Weaknesses

(1) Over-segmentation with SAM — could one explicitly filter out noisy captions that correspond only to object parts instead of relying on the MIL to do much of the heavy lifting? For instance, could one evaluate the image features of object-parts with full-objects to automatically recognise (and filter out) over-segmentations? (2) Noisy knowledge expansion with LLMs. The authors are aware of this, and discuss this. The MIL loss is introduced to handle this noise. Nevertheless, this still means that the approach is training the model with data that we know is quite noisy. Indeed, it results in performance improvement. But might be nicer to not augment w/ so much noise. This is a limitation, and it would be nice to explore a better way to augment this “knowledge”. It’s an open question

Questions

(1) How much worse is these noisy predicted captions algorithm compared to when you may use GT captions? I.e., what may be the upper-bound of this method if we consider high quality captions? I'm not sure if there's a good way to evaluate this. I'm just curious, so I'm raising this point. (2) What about potential improvement in performance with dataset size? Since the approach uses existing pretrained models, at some point, one would expect the performance to saturate as one keeps increasing the size of the unlabeled dataset.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the paper discusses limitations.

thanks! 6->7

thanks for the helpful response! increasing my score from 6->7 as I think you've addressed all my main concerns.

Thanks!

Thanks for the thorough response! Many of my comments have been addressed, thanks for the updates. I will raise my scores accordingly. That scaling plot is quite promising!

Thanks for the response

Thanks the authors for the response. I re-examine some details of the paper, somehow, this work still looks incremental, for example, how to differentiate it with one of important baselines (SVLC)? This work is built on the top of SVLC, (and its codebase) which is as one of the components in the paper, and the major difference is built on the top of BLIP2 backbone. It is reasonable for the gain, if comparing this work with SVLC baseline in the table 1.

Thank you for the clarifications!

One of my primary concerns on reading the paper were about lack of details on key parts of the proposed process (ex. the SAM expander) - the explanation in section 3.2 (ll.179-190) was unclear and not sufficient to understand how the segments were used for generating captions or why they would help on relations. The authors have provided further clarifications on this and I hope they will be included in the revision of the manuscript. On the caption quality analysis, it was previously unclear what was being displayed in Fig 3(a), and the measurement of quality in terms of only clip score, was insufficient to verify whether this step was indeed valuable. I thank the authors for the human evaluation and am glad to see that there is a clear improvement in the caption quality. In addition, would it be possible to also include an analysis on the vocabulary of the generated captions - if it increases/decreases compared to the original captions, in terms of nouns, attributes and relations? This would also help to get a better understanding of why the generated captions contribute to the improved compositional understanding. On the evaluation of BLIP2: Table 1 caption states "BLIP2’s heavier encoder-decoder architecture gives it some advantage on the VL-checklist evaluation. Still, we outperform it by a large margin." This is misleading and would make the reader (such as myself) assume that it was evaluated in an encoder-decoder manner. And to clarify, I did not suggest comparison with the paper I cited, which I am aware, appeared after the NeurIPS deadline - but merely wanted to point out that even a blind LM only model can improve performance by a lot on ARO - so the results from this paper could be put in context of that. However, I agree with the authors that improving compositional reasoning on encoder-only (dual encoder models like CLIP) is valuable and this work shows how to do this. I ask the authors to include a discussion on this, to make it clear to the reader that the proposed improvements are mainly catering towards this type of model, and might not hold significance when applied to models having encoder-decoder setups unless they have results to support that their proposed methods also help for these other kinds of VLMs. The fine-tuning experiments are also great to have, and make the paper's claims stronger.