Scaling Dual Stage Image-Text Retrieval with Multimodal Large Language Models

Abstract

Multimodal large language models (MLLMs) constructed powerful potential through generative training in many downstream tasks. However, a significant performance gap remains in zero-shot multimodal retrieval tasks when compared with contrastively trained dual-encoder models. Existing dual-encoder models have limitations, such as a lack of fine-grained sentence understanding and extremely short input text length constraints. Our research focuses on enhancing the embedding capabilities of generatively trained models to address the limitations of traditional image-text multimodal models, particularly in multimodal retrieval tasks. We specifically investigate the generation of high-quality image-text embeddings through in-context learning and contrastive finetuning. Our novel approach utilizes and optimizes the capabilities of MLLMs, including the design of prompts and examples for in-context learning. Performance optimization using contrastive learning techniques, even under limited computational resources, leads to significant improvements in both multimodal zero-shot retrieval and unimodal tasks. We enhance the MLLM’s ability to discern incorrect semantics by selecting challenging negative pairs from existing datasets. Our results demonstrate the enhanced capabilities of generatively pretrained models on multiple zero-shot image-text retrieval tasks, significantly surpassing the performance of existing dual-encoder models. This code is on Github.