MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

**Re: (Weakness1) The proposed methods for hard sample mining and re-ranking using large multimodal models are straightforward and intuitive, and may lack some novelty.** We thank you for your comment. The main contribution of the work is that we are the first to build a universal multimodal retriever, which reaches state-of-the-art retrieval effectiveness on both multimodal and text-only retrieval while most of the existing work only focuses on one or just a few limited retrieval tasks. Additionally, it is an important finding that MLLMs can be prompted as a strong zero-shot reranker for those complex and challenging tasks involving multimodal queries since such retrieval training data is hard to collect. We believe this study and findings are novel and provide valuable insights to the community. **Re: (Weakness2) There is no direct comparison with similar baseline models, such as E5-V, on the M-BEIR dataset.** We thank you for the insightful questions and agree that we should make a comparison with the concurrent work. Please see the general response. **Re: (Q1) In Table 1, why is the M^rand column for MTEB Text Retrieval empty? Are these models not evaluable on MTEB? I believe that comparing M^rand with M^hard would more effectively demonstrate the modality bias issue and the effectiveness of the proposed method.** We thank you for the insightful suggestion. We have ran inference for $M^{\text{rand}}$(LLaVa-P) and $M^{\text{rand}}$(NVEmb) to compare with their $M^{\text{hard}}$ counterparts as shown below. We observe that $M^{\text{hard}}$(NVEmb) show slight degradation compared to $M^{\text{rand}}$(NVEmb), which is expected, since the hard negative mining aims to address the text retrieval modality bias in NVEmb. On the other hand, interestingly, a reverse trend can be observed in LLaVa-P. We hypothesize that $M^{\text{rand}}$(LLaVa-P) has no such worse text retrieval bias issue compared to $M^{\text{rand}}$(NVEmb); thus, still improve the general retrieval capability in hard negative mining training. |MTEB Text Retrieval |$M^{\text{rand}}$(LLaVa-P) |$M^{\text{rand}}$(NVEmb) |$M^{\text{hard}}$(LLaVa-P) |$M^{\text{hard}}$(NVEmb) | | ----------- | ------------- | ------ | ------ |------ | nDCG@10| 40.8| 51.6| 46.4| 49.7| **Re: (Q2) In Table 4, why does re-ranking show limited improvement for knowledge-related multimodal retrieval datasets like OVEN and InfoSeek? and even a decline in performance on some tasks (e.g., task1 and task3)? How do the authors explain this phenomenon, and could it be related to the modality of the retrieval task?** We thank you for the insightful questions. If we understand correctly, you want to ask why the zero-shot re-ranking does not show significant effectiveness improvement in Table 3 (task 6 and 7). We hypothesize that the fine-tuned MLLM retrievers also learn some task-specific knowledge compared to the zero-shot reranker. Thus, when observing the results in Table 4, we can see a significant gap between the retrieval and reranked results, where all fine-tuned retrievers are not specifically fine-tuned on the task of CIRCO. We will add the discussion and our hypothesis in the revised manuscript. **Re: (Q3) In line 343, the authors mention that MLLM-based retrievers tend to retrieve text over images. Could this be related to the sampling of negative samples during training? If there are more image samples in the dataset, this could lead to a higher proportion of images as negative samples, which could further induce bias. I suggest conducting a statistical analysis on the modality distribution of the data samples.** We thank you for the insightful questions and agree that we should investigate the negative sampling distribution in our random negative sampling approach. The below table reports the total number of documents in text, image or text-image interleaved modalities in train set. We observe that while randomly sampling negatives, text is the most likely sampled modality but MLLM-based retrievers still tend to retrieve the documents with text modality. This means that text modality bias is not resulted from random negative sampling. This analysis provides additional evidence of text retrieval bias on MLLM-based retrievers. We thank you for the helpful suggestions and will add the analysis in our Appendix. ||Text |Image |Interleaved Text-Image | | ----------- | ------------- | ------ | ------ | |Number| 568,757 | 364,903 | 399,130 | |Percentage| 42.6% | 27.4% | 29.9%|

Many thanks for your comments and feedback. We discuss your raised points in the following. **Re: (Weakness1) The paper claims that existing retrieval models typically addressed search scenario where retrieval task are fixed and only a single modality is supported for both queries and retrieved results, but a series of retrieval models for multimodal retrieval are listed in related work.** We thank you for the comments. Although we list a series of multimodal retrieval work in related work for a comparison, most of the work only focuses on limited retrieval scenarios. For example, Liu et al.[1] only explore multimodal retrieval in a single task, WebQA (which is one of the subtask in our evaluation). The more recent work[2][3] also only develops multimodal retrievers for limited tasks (e.g., MSCOCO and Flickr). Furthermore, all the work simply assumes that all the documents in the corpus are in the same modality while our work considers the most practical retrieval scenario, where documents with diverse modalities are included. We highlight and address the issue of modality bias, and build a SoTA universal retriever under such challenging scenarios. We will revise our manuscript to clarify the claim. We also provide a more comprehensive empirical comparison with the recent models in general response as another supporting evidence. [1] [Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval.](https://arxiv.org/pdf/2209.00179) [2] [Jina CLIP: Your CLIP Model Is Also Your Text Retriever](https://arxiv.org/abs/2405.20204) [3] [E5-V: Universal Embeddings with Multimodal Large Language Models](https://arxiv.org/pdf/2407.12580) **Re: (Weakness2) The paper claims that it is the first work to explore prompting MLLMs for zero-shot reranking, but as far as I know, the work [a] has used MLLMs for reranking, and the differences should be explained in detail.** We thank you for providing the valuable reference[1]. We agree that a comparison with the reference work should be made. There are two main differences between our approaches. First, the reference paper leverages query likelihood[2] (i.e., the probability of generating the text query given the image as answer) for reranking, which is only suitable when queries are in text format. In contrast, our prompt-based approach leverages MLLMs’ instruction following capability and formulates any multimodal retrieval tasks into instruction following tasks. Second, the reference paper is mainly to improve the zero-shot retrieval approach while we explore zero-shot reranking upon the strong retrievers which have been fine-tuned on diverse training data for multimodal retrieval. We have added the comparison in our Related work (see our Related Work in the revised manuscript) and also see the potential to extend the query likelihood approach to more diverse multimodal retrieval tasks (see lines 534--536 in the revised manuscript). [1] [Unified Text-to-Image Generation and Retrieval](https://arxiv.org/pdf/2406.05814) [2] [Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking](https://aclanthology.org/2023.findings-emnlp.590/) **Re: (Weakness3) Some expressions are confused. For example, in Line 308 of Page 6, “we initialized retriever using the pretrained model”, does the “retriever” refer to the model to be fine-tuned with hard negatives?** Thank you for the careful examination and your understanding is correct. While fine-tuning MLLMs with hard negatives, we retrain the models (i.e., initialize with pre-trained MLLMs) rather than continuously fine-tuning the $M^{\text{rand}}$ models. We have revised our manuscript accordingly (see lines 308--309 in the revised manuscript). **Re: (Weakness4) It is not easy to understand Figure 1, where the task definition of Task 1-13 is missing. The relationship between the upper left and lower right sub-figure is not clearly illustrated.** We thank you for the helpful suggestion. We have revised our Figure 1 and caption to clarify the task definitions and the illustrations of retriever fine-tuning and zero-shot reranking (see Figure 1 in the revised manuscript). Continue...

We really enjoy the discussion and thank you for providing the valuable suggestions to improve our manuscript.

Many thanks for your comments and feedback. We discuss your raised points in the following. **Re: (Weakness1) Unfortunately, the authors are not the first to use MLLM for re-ranking. It is recommended that the author faithfully investigate related research, e.g., [1]. The authors should narrow down the scope of the study, i.e., composed image retrieval.** We thank you for pointing out the valuable reference, MERLIN[1]. We would consider this work as query rewriting or pseudo relevance feedback[2] in dense retrieval rather than prompt MLLMs for zero-shot reranking. First, the reference paper uses the embeddings from Google Multimodal API for both retrieval and reranking, which is a close-source model. Thus, we are not certain whether the API models used for reranking have been fine-tuned for retrieval tasks or not, while in our zero-shot reranking, we use LLaVa-Next, which is only fine-tuned on instructions without learning the tasks of retrieval. Nevertheless, we believe the approach in MERLIN can be further added to our retriever, UniEmb, to refine the query embedding with multiple search iteration to improve the final retrieval effectiveness, which is also a promising future direction and we will cite the reference paper in our future work. Also, we agree that we should fully investigate the related work for MLLM reranking to clarify our contribution (see our Related Work in the revised manuscript). Specifically, our approach is more close to prompt instruction fine-tuned models for zero-shot reranking[3][4], which has been previously explored in text ranking with LLMs but as far as we know, has not been comprehensively explored yet in multimodal retrieval with multimodal LLMs. [1] [MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline.](https://aclanthology.org/2024.emnlp-industry.41/) [2] [Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls.](https://dl.acm.org/doi/10.1145/3570724) [3] [Is ChatGPT good at search? investigating large language models as re-ranking agents](https://aclanthology.org/2023.emnlp-main.923.pdf) [4] [A setwise approach for effective and highly efficient zero-shot ranking with large language models](https://arxiv.org/abs/2310.09497) **Re: (Weakness2) The author's claim in line 184 should be verified experimentally using either very large k or very small k.** We thank you for the helpful suggestions and agree that a clear explanation on the claim in line 184 should be given. In this paper, we directly borrow the previous hard negative mining study on text retrieval training from [1][2], which shows suboptimal retrieval effectiveness when mining hard negatives from too large and small top-$k$ candidates. Additionally, we also examine the retrieved results from our $M^{\text{rand}} (NVEmb)$ (which we use for hard negative mining for $M^{\text{hard}} (NVEmb)$) on the training set. Furthermore, we observe that the top-1 accuracy is above 50% in many query sets, such as WebQA, OVEN and INFOSEEK. This means when setting $k=1$, above 50% of the mined negatives are false negative while setting large $k$, the mined negatives would be similar to randomly sampled negatives from the corpus. We have added the explanation to our manuscript (see lines 203--205 in the revised manuscript). We agree that the optimal $k$ is case dependent and it is worth conducting a more comprehensive study on hyperparameter search in multimodal retrieval training, which we leave for future work. [1] [APPROXIMATE NEAREST NEIGHBOR NEGATIVE CONTRASTIVE LEARNING FOR DENSE TEXT RETRIEVAL](https://openreview.net/forum?id=zeFrfgyZln) [2] [NV-Retriever: Improving text embedding models with effective hard-negative mining.](https://arxiv.org/pdf/2407.15831) **Re: (Weakness3) Line 306: The author's fine-tuned batch size for the two types of models is different. Will this affect the effect of mining negative samples?** We thank you for the insightful question. We set different batch size at the two stages is because at the second stage, we pair each query with additional hard negative; thus, the total size of in-batch negatives is $2 \cdot |\mathcal{B}| - 1$ compared to the total size of in-batch negatives at first stage $|\mathcal{B}| - 1$, where $|\mathcal{B}|$ is batch size. Thus, to make the total size of in-batch negatives the same at both stages, we use $2 \times$ larger batch size at first stage compared to the second stage. We revised our manuscript to clarify the reason behind the choice (see lines 211--215 in the revised manuscript). Continue...

Many thanks for your comments and feedback. We discuss your raised points in the following. **Re: (Weakness1) The details of the proposed method are not clear and confusing. In sec. 4.1.1, it seems random negatives are denoted as c_i^{\times} symbol, while the generated triplets use c_i^{\times} again, does it mean positive sample or random negative?** We thank you for pointing out the unclarity in our manuscript. To clarify, in the setting of training with random negatives, we pair each query in a mini batch with a positive; i.e., $((inst_1,q_1), c_1^+), ((inst_2,q_2), c_2^+), \cdots \((inst_{|\mathcal{B}|}, q_{|\mathcal{B}|}), c_{|\mathcal{B}|}^+\)$. While conducting contrastive training using Eq (1), we have in-batch documents from all the positive documents $\mathcal{D} = (c_1^+, c_2^+, c_{|\mathcal{B}|}^+ )$. Thus, all the documents in $\mathcal{D}$ except for $c_i^+$ are considered randomly sampled negatives for $(inst_i,q_i)$. As for hard negative mining, we pair each query with a positive and hard negative $((inst_i,q_i), c_i^+, c_i^-)$. Thus, in the setting of hard negative mining, we have in-batch documents from all the positive and negative documents $\mathcal{D} = (c_1^+, c_1^-, \cdots, c_{|\mathcal{B}|}^+, c_{|\mathcal{B}|}^-)$. Similarly, all the documents $D$ except for $c_i^+$ are considered negatives for $(inst_i,q_i)$, including one hard negative $c^-_i$ and all the positives and negatives from other queries as random negatives. We have revised our manuscript to make it more clear (see lines 195--215 in the revised manuscript). **Re: (Weakness2) In sec. 4.2, the authors prompt LLaVa-Next for reranking, while not introducing the detailed MLLMs adopted in sec. 4.1. Are there two multimodal LLMs needed for the proposed method?** We thank you for the suggestions. To clarify, we adopt the same MLLM backbone, LLaVa-Next. While retrieval, we explore fine-tuning MLLM as a retriever while reranking, we directly prompt the pre-trained MLLM as a zero-shot reranker. We have revised our manuscript (see lines 160--161 and lines 235--237) and Figure 1 in the revised manuscript to make it more clear. **Re: (Weakness3) It will be better to provide the framework figure for the whole method.** We thank you for the suggestions. We have revised our Figure 1 to illustrate the whole framework of our paper, including MLLM retriever fine-tuning and prompt-and-reranking with pre-trained MLLMs (see Figure 1 in the revised manuscript). **Re: (Weakness4) In Table 4, it is not clear which results denote the reranking results, it seems the last row does not outperform the fifth row.** We thank you for the clarifying question. In Table 4, the first column denotes the retrieval result for each multimodal retriever (from MagicLens to our UniEmb retriever) and the second column denotes the zero-shot reranking result upon the top-10 candidates from each retriever. Thus, when comparing the first and second column, we can see that the zero-shot reranker improves over all the retrievers. We revised Table 4 to make it more clear to readers. **Re: (Weakness5) As a retrieval method, the efficiency of retrieval time and memory is as important as effectiveness. The authors should analyze the efficiency of the unimodal retrieval mechanism.** We thank you for the insightful suggestions. We measure the index storage required for the 5.6M document from the M-BEIR datatset. As for retrieval latency, we measure the latencies of query encoding and vector search. For query latency, we randomly sample 100 queries from each test query pool in the 16 M-BEIR tasks and measure per query encoding and vector search latency with a batch size of 1. Since query encoding latency is varied with query length, we report the latency at 1th, 50th and 99th percentiles. We have added the measurement and detail in Appendix A.2 in the revised manuscript. |Retriever |storage (GBs) |Latency(ms) |Latency(ms)| | ----------- | ------------- | ------ | ------ | | |**Index** | **Encoding** (1st / 50th / 99th perc.)| **Vector search** | | $\text{CLIP}_{\text{SF}}$| 16| 26 / 27 / 39|6| | UniEmb |86| 81 / 194 / 203| 33|