Improving CLIP Training with Language Rewrites

Summary

CLIP uses data augmentation for image inputs but neglects the diversity of texts associated with the same image. To overcome this limitation, this paper introduces Language augmented CLIP (LaCLIP), a simple yet highly effective approach that enhances CLIP training through the startegy of language re-writing. Extensive experiments conducted on CC3M, CC12M, RedCaps, and LAION-400M datasets demonstrate the significance of LaCLIP, which outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M in terms of ImageNet 0-shot acc. without imposing additional computation or memory overhead during training.

Strengths

Overall, this is a good paper as it has notable contributions to the community. Vanilla CLIP often exhibits an imbalance where the image encoder is strong while the language encoder is comparatively weaker. It is partially because CLIP has very diverse image inputs yet lacks proper data augmentation strategies for language. This work presents a very interesting solution, i.e., language re-writing with a third-party language model or human efforts. The motivation behind the paper is commendable, as it is addressing a very important problem for vision-language pre-training. The method LaCLIP is simple and easy to follow. The experiments are extensive, and the results are very significant.

Weaknesses

My major concern lies in whether LaCLIP could be regarded as a method of knowledge transfer. Specifically, while augmenting the captions by a large language model, the CLIP text encoder actually distills some knowledge from that. So, I am wondering if your LaCLIP outperforms vanilla CLIP simply because LaCLIP has a stronger text encoder. To ablate this problem, we can first load or distill from a pre-trained language model as CLIP's text encoder and then perform its training process. There are also some minor drawbacks: LaCLIP relies on a well-pretrained language re-writer such as ChatGPT and LLaMA, which incurs heavy computing cost for language data augmentation and prevents end-to-end training.

Questions

1. How is the difference between your LaCLIP and the paradigm mentioned in "Weaknesses"? 2. Does a stronger LLM always yield better LaCLIP performance?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

See "Weaknesses".

Summary

This paper proposes a new language augmentation method for training CLIP. Specifically, a large language model is prompted, by a few examples, to generate rewrites for existing texts in image-text paired datasets. Gains are demonstrated on CC12M, RedCaps, LAION-400M, etc. compared with CLIP and SLIP.

Strengths

1. The proposed solution looks simple, taking good advantage of existing pretrained LLMs. 2. Extensive training and evaluation experiments on multiple publicly available datasets and backbones, within a reasonable computation budget. 3. Good ablations on immediate variants of augmentation strategies and ICL strategies.

Weaknesses

1. Missing comparison with or discussion on relations to existing text augmentation methods applied to CLIP training, e.g. DeCLIP [1] introduced self-supervision and multi-view supervision which generated 2x2 pairs with both image augmentation and text augmentation such as synonym replacement, random swap, and random deletion. LaViLa [2] explored augmenting with a rephraser and even re-captioning with vision conditioned large language models. 2. Text augmentation might be addressing the issue of limited text data in image-text paired datasets. How would text augmentation compare with finetuning or frozen text encoder pre-trained on large scale text data? e.g. with BERT, ROBERTA, or MPNET embeddings, instead of learning from scratch on image-text paired datasets. 3. The current method uses ICL on LLaMA, how about instruction tuned models, do we still need the three examples? 4. It is also interesting to see examples of how language augmentation helped the learning of visual features. What are the examples that were corrected? 5. Text encoder size worths ablating, since the proposed solution focuses on text augmentation. 6. Scaling is unconfirmed with a large dataset and a large backbone at the same time, but this is not doable given limited compute. [1] Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm. ICLR 2022. [2] Learning Video Representations from Large Language Models. CVPR 2023.

Questions

It is preferred if there are existing results addressing the questions in the weaknesses section, but there is no need to do extra training due to the computation needed. There are related works on arxiv. Discussion is recommended but not required. [1] A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision. https://arxiv.org/abs/2112.13884. [2] Improved baselines for vision-language pre-training. https://arxiv.org/abs/2305.08675.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

This paper introduces a simple text augmentation strategy to train vision-language models. Given an image-caption dataset, the core idea is to use an off-the-shelf LLM to "rewrite" image captions. Since LLM outputs do not necessarily read like image captions, authors use in-context learning -- input a few example caption rewrites to the LLM in-context to generate the rewrite. The authors use this augmentation to train CLIP, which they call "Language Augmented CLIP" (or LaCLIP). LaCLIP significantly outperforms CLIP on multiple downstream tasks at various training data/model sizes.

Strengths

I think this paper matches the quality of a typical publication at the NeurIPS conference. I recommend acceptance; it is relevant to the conference audience and will spur exciting discussion in the community. Below I highlight the main strengths of the paper: 1. **Simplicity:** The proposed method is conceptually simple and empirically powerful. It improves the performance of contrastive image-text models like CLIP and SLIP on many downstream vision tasks. 2. **Proposed approach has no 'online' training overhead:** Language rewrites can be performed once for the entire training dataset 'offline' and saved to disk. During training, the only extra overhead is loading all candidate captions per image (a few extra lines of text), and doing a random number generator to select one candidate caption. This overhead is negligible — considering this, the empirical improvements are very appealing. 3. **Experimental thoroughness:** This paper is an excellent example of an empirically thorough study. The authors are interested in answering a single general question: How much can text augmentation benefits current CLIP-style models? The authors do an excellent exploration to answer this question: - Many design choices for language rewrites are experimented with, even a baseline that sources meta-prompts from human annotators! - This augmentation is plugged in with two approaches (CLIP and SLIP) to show its "drop-in" benefits. - Datasets and model architectures of different sizes are used to showcase the scaling behavior of this approach. - Empirical evaluations cover a variety of downstream classification tasks (zero-shot, linear probing, few-shot) and 15+ datasets. 4. **Excellent clarity in writing and presentation:** All technical details for empirical analysis are well-stated and easy to follow. The main paper and supplementary material have adequate implementation details to aid reproducibility. All result tables are neatly organized to highlight the central messages to the reader efficiently.

Weaknesses

I have a few questions/concerns which I believe should be addressed or acknowledged, and other suggestions (optional) to improve the paper. 1. **Writing can be adjusted to decouple the approach from CLIP:** The proposed approach is at its core, a data augmentation strategy for methods that train with image-text pairs. This broadly encompasses methods that perform generative training, like image captioning (e.g. VirTex, BLIP) and masked language modeling (e.g. ICMLM). The writing can be slightly reworded to convey this, along with mentioning that in this paper only one instantiation is considered -- contrastive models like CLIP. 2. **Connection with prior works like BLIP?** BLIP and other such works perform iterative training through language rewrites, using a captioning model to _replace_ training captions. The proposed approach uses language rewrites as text augmentations -- I believe that making this connection in the paper is useful. 3. **Proposed approach lacks a mechanism to remove noisy captions:** This approach expands the set of candidate captions per image through language rewrites. However, the original caption in training data may be noisy and uninformative, which is common in web-scale datasets. This approach does very little to remove such captions, and may not generate a semantically relevant caption through rewrites because the rewrites do not condition on the input image.

Questions

I have one question related to the weaknesses I mentioned: Have the authors tried training non-contrastive vision-language models with this training strategy?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have discussed the limitations and broader impact of their method in the appendix. I agree with the authors' assessment and believe that the discussion is sufficient. However, I urge the authors to move it to the main paper and instead transfer any side experiments or implementation details in the appendix.

Summary

This paper introduces a straightforward yet highly effective language augmentation technique for the foundational vision-language pre-training model, CLIP. The authors focus on exploring the in-context learning capabilities of large language models, such as LLaMaA-7B, to produce four distinct rewrites for each text sample in the dataset. These rewrites are generated using ChatGPT, Bard, COCO, or Human meta-input-output text pairs, making the augmentation process relatively simple. On the experimental front, the authors present compelling evidence of strong transfer performance across multiple benchmarks, which is unsurprising given the rationality of informative text data augmentation. However, the paper could be even more impactful if the authors provide more thorough calibrations as suggested in the weakness section. Overall, this work contributes a valuable language augmentation method to the CLIP paradigm, and its potential recognition within the broader community could be further solidified by addressing the weaknesses and providing additional calibrations on why and how the proposed method work.

Strengths

+ This work exhibits a clear and compelling motivation, along with a straightforward and elegant solution. + The experimental results presented in Table 1, Table 2, and Table 3 demonstrate the strength and efficacy of the proposed approach. + The detailed descriptions provided in the paper render it highly accessible and easy to follow for readers.

Weaknesses

The authors of this paper primarily focus on reporting impressive results without conducting a thorough analysis of why their proposed method performs better. To enhance the clarity of their work and provide a deeper understanding of their approach, several key aspects need to be addressed. - First, it is essential for the authors to include the training loss and validation loss curves in their analysis. This will help readers discern whether the proposed method improves optimization (by achieving smaller training and validation losses) or generalization (by achieving a smaller validation loss only). - Second, the main difference between the source description and the rewritten description lies in enriching the presented concepts with more detailed descriptions. However, this approach also faces the risk of hallucination, potentially generating descriptions that do not accurately reflect the facts. The authors should thoroughly analyze this issue and propose methods to alleviate it, rather than relying solely on hand-crafted instructions to guide the large language models (LLMs). - Furthermore, there is a glaring absence of critical ablation experiments in the paper. To provide a comprehensive evaluation of their method, the authors should consider the following groups of ablation experiments: 1. **The influence of the length of the rewritten description**: The authors could attempt to generate rewritten descriptions of varying lengths, such as 1.5x or 2x, or even up to 5x longer than the original descriptions. By reporting comparison results and analyzing the outcomes, readers can gain insights into the impact of description length on the model's performance. 2. **The influence of the text encoder model scale**: The authors currently opt for the smallest text encoder from CLIP, following previous work. However, it is vital to explore whether the rewritten descriptions may require a larger text encoder, especially for significantly longer rewrites. Providing detailed comparison results will offer valuable insights into the significance of the text encoder model's scale. By addressing these critical points, the authors can strengthen their work's analysis, transparency, and overall contribution to the field.

Questions

The authors should provide more analysis and discussion on why the proposed method performs better except for reporting straightforward comparison results.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Good rebuttal

Thanks for your detailed responses. The author is encouraged to add the rebuttal contents to the main paper in the future. In light of the authors' response, I have adjusted the original rating from "Borderline Accept" to "Weak Accept."

Looking forward to follow-up discussion

Dear Reviewers, We would like to thank you again for your effort and insightful comments! As the deadline for the discussion period is approaching, we would like to inquire if there are any additional clarifications or experiments that we can provide. We would greatly appreciate it if you could consider favorably updating the review if our response successfully addressed your concerns. Once again, we extend our heartfelt appreciation to the reviewers for their time and insightful suggestions. All the additional discussions will be incorporated in the next version of the paper. Best, Authors

Rebuttal follow up

Dear Reviewer YRKL, Thank you so much for liking our work and sharing the insightful comments again! As we approach the final day of the discussion period, we are reaching out to you to ensure that our rebuttal effectively addressed your concerns. Moreover, we would love to provide further clarification or conducting additional experiments, should you deem it necessary. 
We will follow your invaluable suggestions to revise our paper and include all of the additional disscusion and experiment insights from the rebuttal into the main paper. We will be meticulous in incorporating all discussions on limitations into the main paper as well. We firmly believe that the inclusion of these elements will undeniably fortify the strength of the paper. Once more, we extend our heartfelt gratitude for your dedication and the time you've graciously devoted to the review of our work! Please don't hesitate to let us know if there's any additional questions or suggestions! Best Wishes, Authors

Rebuttal follow up

Dear Reviewer FjR1, Thank you for the positive feedback and the insightful suggestions again! We have conduct some **additional experiments** to further resolve your concerns with regard to **pre-trained text encoder**. Here is a summary of our new results: In addition to our discussion in the rebuttal, we directly compare the pre-trained text encoder setting in our exact experiment setup on CC12M, to further show the advantage of LaCLIP comparing to using pre-trained text encoders. We replaced the text encoder and tokenizer with pre-trained BERT-Base model, and kept all other parameters to be the same. We tested both of the 2 suggested setups: *fine-tuning* the whole model and *frozen* BERT weight: < Table S3>. Zero-shot performance comparison between different pre-trained text encoder setups on CC12M | Method | Pre-trained Text Encoder | Text Encoder Freeze | Downstream | ImageNet | |---|:--|:--|:-:|:-:| | CLIP (Vanila) | N/A (from scratch) | No | 38.8 | 40.2 | | CLIP (BERT-Fine-tune) | BERT-base | No | 42.1 | 42.9 | | CLIP (BERT-Frozen) | BERT-base | Yes | 24.5 | 23.2 | | LaCLIP | N/A (from scratch) | No | **46.2** | **48.4** | The observations align with the findings depicted in Figure 3 of LiT [1]. Fine-tuning the pre-trained BERT model exhibits some enhancement in CLIP training performance, whereas retaining the frozen BERT encoder substantially degrades performance. In contrast, LaCLIP consistently outperforms all BERT pre-training configurations, underscoring the necessity for explicit sentence augmentation strategies. [1] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A. and Beyer, L., 2022. Lit: Zero-shot transfer with locked-image text tuning. In CVPR 2022. In our previous rebuttal we meticulously followed the detailed suggestions provided by you, and have added all of the requested experiments, discussions and visualizations to make our work even more comprehensive. We will add all detailed response into the final version of the paper. Since there is only 1 day left in the discussion period, we would like to kindly ensure that the reviewer has seen our response, and eager to know whether there is additional clarification or experiments that the reviewer would like us to offer. We would be extremely grateful if the reviewer could consider favorably updating the review if our response effectively addressed your concerns. Thanks again for the effort and time you have dedicated to our work! Please let us know if you have additional comments or questions. Best Wishes, Authors

Rebuttal follow up

Dear Reviewer qFtF, Thank you for acknowledging the contributions of our work as well as your thoughtful insights once again! In addition to our discussion in the rebuttal, we have done some additional experiments with regard to your previous suggestions on **Pre-trained Text Encoder**, we put our **new reults** here and hope you to find them interesting and convincing: We follow your suggestions and directly compare the pre-trained text encoder setting in our exact experiment setup on CC12M, to further show the difference of LaCLIP comparing to using pre-trained text encoders. We replaced the text encoder and tokenizer with pre-trained BERT-Base model, and kept all other parameters to be the same. We tested 2 setups: *fine-tuning* the whole model and *frozen* BERT weight: < Table S2>. Zero-shot performance comparison between different pre-trained text encoder setups on CC12M | Method | Pre-trained Text Encoder | Text Encoder Freeze | Downstream | ImageNet | |---|:--|:--|:-:|:-:| | CLIP (Vanila) | N/A (from scratch) | No | 38.8 | 40.2 | | CLIP (BERT-Fine-tune) | BERT-base | No | 42.1 | 42.9 | | CLIP (BERT-Frozen) | BERT-base | Yes | 24.5 | 23.2 | | LaCLIP | N/A (from scratch) | No | **46.2** | **48.4** | The observations align with the findings depicted in Figure 3 of LiT [1]. Fine-tuning the pre-trained BERT model exhibits some enhancement in CLIP training performance, whereas retaining the frozen BERT encoder substantially degrades performance. In contrast, LaCLIP consistently outperforms all BERT pre-training configurations, underscoring the necessity for explicit sentence augmentation strategies. [1] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A. and Beyer, L., 2022. Lit: Zero-shot transfer with locked-image text tuning. In CVPR 2022. With 1 day remaining in the discussion phase, we would like to confirm if our response successfully addressed your concerns. Furthermore, we would love to provide additional clearification or experiments that you'd like to see! We will integrate our rebuttal based on your suggestions into the final version of the paper, including the distinctions between LaCLIP and pre-trained text encoders, results with stronger LLMs, and discussion on the computational costs. The inclusion of these aspects is poised to significantly enhance the quailty of the paper! Your dedication of time and effort in reviewing our work is truly appreciated. Please let us know if you have any additional comments or questions! Best Wishes, Authors