Summary
This paper proposes a TripletCLIP, a pre-training framework aimed at enhancing compositional reasoning task. It computes the NegCLIP objective separately for each, with the image serving as the anchor in each instance. For training TripletCLIP, such triplet data is constructed as follows: Hard negative captions are generated from the original LaCLIP captions using a large language model (LLM) through in-context learning mechanism. Hard negative images are created from text-to-image diffusion models. The experiment section compared various hard negative (HN) based objectives and demonstrated the superior performance of the proposed TripletCLIP mechanism across a wide range of benchmarks: compositionality, zero-shot retrieval, and classification.
Strengths
[Originality and Significance]
This paper addresses both model and data aspects in the context of pre-training CLIP. Given the challenges of synthesizing hard negative texts and particularly images at scale, the proposed work for constructing triplet data is likely to be valuable to the research community. On the model side, the proposed hard negative objective in pre-training, though simple, consistently enhances performance across a variety of tasks, including compositionality, retrieval, and classification.
[Quality and Clarity]
The experiments are well-designed and meticulously verify the claims made in the paper, demonstrating the necessity of such design choices. Some additional baselines, beyond the default LaCLIP, are included to demonstrate the effectiveness of TripletCLIP. The overall presentation was clear.
Weaknesses
- [W1] The effectiveness of the TripletCLIP objective is less convincing. Although the paper considered some hard negative (HN)-based baselines such as NegImage and NegCLIP++, it did not compare with HN baselines that jointly consider both HN text and image in the loss calculation, such as [1, 2, 3, 4, 5]. I understand that some works are concurrent, but it would be more comprehensive to see whether the design of the TripletCLIP objective is optimal, either quantitatively or conceptually. In addition, it is necessary to note the 'previous' work that generates hard negative images to enhance compositional reasoning tasks [2, 3]. Accordingly, the sentence from [L136-137] needs to be adjusted.
- [W2] Though there are consistent improvements, the absolute level of performance is relatively weak compared to the 'default' CLIP models from the CyCLIP [1] and ALIP [6] papers (in terms of zero-shot imagenet and retrieval tasks). This may be due to the reduced computational resources during pre-training, which limits the significance of the results in the context of pre-training. The only possibility left, and neither the triplet data nor the pre-trained model shows the evidence of benefiting further downstream tasks in applications such as advanced VLMs like LLaVA, or text-to-image generation, as noted in [L70-72].
- [W3] Connected to [W2], if only a single GPU is used for pre-training, it would be an attractive alternative to fine-tune the pre-trained CLIP model with the (subset of) triplet data. It's clear that the proposed methodology remains valid, and it would be interesting to see whether employing triplet data with the proposed hard negative objectives proves superior compared to other compositional reasoning methodologies in the context of fine-tuning [7, 8], specifically on both compositional reasoning and retrieval/classification tasks. If the authors wish to focus solely on the pre-training context, referencing other previous work discussing both pre-training, as well as fine-tuning is missed [9].
- [W4] It is suggested to consider including additional compositional benchmarks for evaluation that consist of counterfactual image-text pairs, similar to the Winoground style: EqBen [2], COCO-Counterfactuals [10], MMVP-VLM [11], and SPEC [4], for comprehensiveness.
In summary, the main concerns are the missing comparisons with hard negative (HN) objectives that incorporate both image and text, and the limited significance due to small-scale pre-training. Fine-tuning could be considered as an alternative approach to address these issues. Superiority over previous fine-tuning methodologies can further increase the significance. Meanwhile, I believe the large-scale triplet data constructed in the paper is valuable and could promote further studies in various aspects.
---
References
[1] Goel et al., CYCLIP: Cyclic Contrastive Language-Image Pretraining, in NIPS 2022.
[2] Wang et al., Equivariant Similarity for Vision-Language Foundation Models, in ICCV 2023.
[3] Sahin et al., Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining, in WACV 2024.
[4] Peng et al., Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding, in CVPR 2024.
[5] Singh et al., Learn “No” to Say “Yes” Better: Improving Vision-Language Models via Negations, in arXiv preprint 2024.
[6] Yang et al., ALIP: Adaptive Language-Image Pre-training with Synthetic Caption, in ICCV 2023.
[7] Doveh et al., Teaching Structured Vision&Language Concepts to Vision&Language Models, in CVPR 2023.
[8] Doveh et al., Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models, NIPS 2023.
[9] Singh et al., Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality, in EMNLP 2023.
[10] Le et al., COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs, in NIPS Dataset and Benchmark Track, 2023.
[11] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, in CVPR 2024.
Questions
- Missing citations [L17-19]; image classification [CITATION], segmentation [CITATION], and image-text retrieval [CITATION]
- From [L180], it needs referring to the proper section in the appendix
- From [L227], what is the meaning of the 'real' hard negatives?