Summary
This paper introduces LotCLIP, which enhances CLIP’s capability to understand long texts. It highlights that merely increasing the length of texts (i.e., context length) is not beneficial as it adversely impacts the understanding of short texts (i.e., image classification). To mitigate this trade-off, additional learnable corner tokens are integrated into the text encoder transformer. Moreover, the attention mask is modified so that interactions between corner tokens are restricted, ensuring diversity in the output feature embeddings. The image encoder and text encoder are initialized from pre-trained models and further trained in a LiT manner. LotCLIP is trained on a self-constructed image-text dataset of 100M scale long texts using MLLMs. LotCLIP is evaluated on both long-text retrieval tasks and short-text benchmarks such as image-text retrieval and classification.
Strengths
- This paper addresses the underexplored yet important problem of enhancing CLIP's ability to understand long texts. It introduces a simple modification to the CLIP training framework.
- The 100M scale long caption data will be valuable for training VLMs with an understanding of long contexts.
- Strong empirical results compared to other baselines such as LiT , CLIP, and SigLIP.
Weaknesses
- [W1] Contributions are unclear. From an architectural perspective, there seems to be no major difference from [1, 2], which introduce additional learnable tokens in the encoders. Is there any special reason or evidence that such a technique is especially beneficial for training with long captions? It is likely to be a general technique that is also helpful for training with short captions. Such learnable tokens, named corner tokens in the paper, are only prepended to the text tokens after the [CLS] token. Corner tokens in different positions, such as after the text tokens, can be further ablated. Would corner tokens also be helpful in vision transformers? More comprehensive analysis around the corner tokens are necessary.
- [W2] Connected to the [W1], the contributions of the data are unevaluable. It only mentions that some image-text pairs from various sources (e.g., CC3M/12M, YFCC15M, LAION, and COYO) are re-captioned using MLLMs such as InstructBLIP, LLaMA, and ShareGPT-4V. No other clear details are provided. Any statistics about the training data are missing. Any details on the MLLMs such as specific architecture and instruction information are completely omitted. For the LAION and COYO datasets, how is a subset selected from the entire scale? Most importantly, no verification step for the extracted long captions is provided. Due to the typical hallucinations of MLLMs, it is unclear how well the obtained long captions align with the original images or original short captions, further complicating the training of VLMs.
- [W3] It is unclear whether the comparison is fair. With the LiT training mechanism, LotCLIP benefits from an ImageNet-pretrained ViT backbone, which shows strong evaluation results on ImageNet compared to other pretrained backbones. It is suggested that other visual backbones pretrained via CLIP or through unsupervised methods such as DINO be tested with LiT training.
- [W4] Training and some evaluation benchmarks seem to overlap, leading to high performance results. For example, ShareGPT has LAION and Conceptual Captions images, which shares the training data. This creates a significant gap in evaluation results compared to DCI, and even between models from the default CLIP and the proposed LotCLIP in ShartGPT evaluation.
- [W5] It is not directly comparable to LongCLIP. Starting from the same pretrained CLIP model, how does LotCLIP perform when fine-tuned on a 1M-scale dataset similar to that used for LongCLIP?
In summary, in the current version, there is no clear evidence of technical contributions and the experimental settings are unclear.
---
References
[1] Darcet et al., Vision Transformers Need Registers, in ICLR 2024.
[2] Lavoie et al., Modeling Caption Diversity in Contrastive Vision-Language Pretraining, in arXiv preprint 2024.
Questions
- The reasoning behind the naming of the constructed data and model is unclear. Why are they named Dora and LotCLIP?
- In the introduction section, the concept of the corner token first appears, but without supporting explanations, which creates confusion.
- No training details for the baseline methods are provided.