Bridge the Modality and Capability Gaps in Vision-Language Model Selection

Summary

This paper considers a zero-shot image classification strategy by selecting the most appropriate Pre-Trained VLM from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. Two challenges, i.e., the “Modality Gap” across two different modalities, and the "Capability Gap" between the VLM’s overall ranking and its ranking for target dataset, hinder the appropriate selection of the different VLMs. To address the challenges, this paper adopts a transportation matrix to capture the relevance between open-source and target datasets, and also uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset. Extensive experiments on image classification datasets demonstrate the effectiveness of the proposed method.

Strengths

Originality: The main novelty of this paper is to utilize Language-Only VLM Selection to address the zero-shot image classification. This is the first time that the Language-Only VLM Selection is applied in the zero-shot scenario where the missing of image data makes it difficult to directly select the VLMs. Quality: This paper has well motivated methods and extensive experiments. Clarity: The paper is well written with clear description and readable figures. Significance: The two key issues of modality gap and capability gap are considered in this paper. In addition, optimal transport-based methods are proposed to alleviate the issues and the experimental results on multiple VLMs and image classification datasets show the effectiveness of them.

Weaknesses

The main weakness is that the proposed VLMs selection strategy relies on Open-Source Datasets’ Data, which restricts the practicability of the proposed method. For the zero-shot image classification task, using the aggregated knowledge from multiple VLMs is ok, but using open-source datasets may violate the zero-shot data restrictions, or even lead to benchmark leakage. In addition, the impact of the open-source datasets is not analyzed. For example, if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks, whether the performance of the proposed method is still remarkable?

Questions

A comprehensive analysis on the impact of the open source datasets may improve the rationality of the proposed method for addressing the zero-shot image classification task.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

This paper does not include the limitation discussion. Practically, at least the issues raised in the weakness section should be considered if they cannot be addressed appropriately in current version.

Summary

With the popularity of Vision Language Model (VLM) research in recent years, many versions have emerged, forming the VLM Zoo. This paper aims to select the most appropriate pre-trained VLM from the VLM Zoo, relying solely on texts of the target dataset without access to images. Two challenges are analyzed, namely, “Modality Gap” and “Capability Gap”. One VLM selection method (SWAB) is proposed: first calculate optimal transportation matrix to capture the relevance between open-source and target datasets; then use this matrix to transfer useful statistics of VLMs from open-source to target dataset for bridging two gaps and enhancing the VLM’s capacity estimation. Experiments are carried out across some VLMs and classification datasets to validate the effectiveness.

Strengths

[+] Due to the differences in training data, architecture and pipelines, existing VLMs do have their own strengths in terms of capabilities. Thus, studying how to better utilize members in VLM Zoo under different scenarios is somewhat valuable. [+] The paper has rich symbols and formalizations, and the space utilization is also reasonable. [+] Some experiments are conducted to demonstrate the effectiveness of components and designs on a diverse set of datasets of image classification.

Weaknesses

[-] Problem rationality. Although each member in the VLM Zoo has their own strengths, relying solely on textual descriptions to find the most suitable VLM (LOVM) seems too demanding. And language is usually compact, which can easily confuse concepts. Why cannot we use one combination of images and texts (or even text-image pairs) for VLM selection, as images are very easy to obtain in real-world scenarios? [-] Selection strategy. Given that the members in VLM Zoo have different strengths and abilities, it means that they have somewhat degree of complementarity. One natural idea is to follow ensemble learning, that is, vote on the results of each member, or select the top-k VLMs’ results for complementary combination. Why only choose the most suitable one? Please provide more comparisons and explanations. [-] Generalization. This paper carries out experiments on many datasets, but all for image classification. Since VLMs excel in handling various tasks, such as detection, segmentation, captioning, and VQA, the generalization on these tasks is still unclear.

Questions

[-] Better visualization. For Fig. 3, there are too many symbols and formulas, increasing difficulty in understanding (the significance of using images is to reduce understanding costs). To this end, the reviewer believes that this paper can be polished better.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

This paper does not discuss the limitations of the work. However, the reviewer seems to find some limitations, such as the unrealistic assumptions in settings and the ability to only handle classification tasks. Please refer to Weaknesses for more details.

Summary

The paper proposes SWAB the modificaiton of LOVM to mitigate the negative impact of two gaps of LOVM: the modality gap and capability gap. The resutls show the effectiveness of SWAB.

Strengths

1. The motivation of this paper is clear and structure is easy to follow.

Weaknesses

1. For the modality gap, the SWAB using open-source datasets, could you please provide how many image samples used in the experiment. And could you please provide the accuracy vs number of open-source image samples? 2. The SWAB seems like a learnig-based model, could you please add the ablition study for no learning based SWAB.

Questions

1. Why adding Gaussian noise to the target generated embedding? 2. How to select weighted parameter $\alpha$ in Eq.9 3. The Auxiliary texts is generated by ChatGPT, could you please test the open-source LLM like LLama2?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No.

Summary

This article focuses on selecting the best model from a visual-language multimodal model zoo for specific downstream tasks without having images of those tasks. It provides a detailed analysis of two challenges faced in this problem — Modality Gap and Capacity Gap. This paper proposes a method called SWAB, which leverages the statistical measures of VLM on open-source tasks based on the category similarity between historical open-source tasks and target tasks, to mitigate the negative impacts of these gaps on Language-Only VLM model selection. Experimental results demonstrate the effectiveness of the proposed method.

Strengths

1.The problem studied in this paper is novel and interesting, with significant practical value. In recent years, the number of open-source VLMs has been increasing, and their zero-shot capabilities have been improving. For practical downstream users, especially those who lack computational resources, have limited funding to label extensive evaluation datasets, or are non-machine learning practitioners with little experience in setting up and evaluating various VLMs, the Language-Only Vision Model selection (LOVM) problem studied in this article is important. 2.This paper presents a clear and well-motivated problem statement. The authors provide a detailed analysis of the negative impacts of the Modality Gap and Capacity Gap on Language-Only Vision Model (LOVM) selection through explanations and experimental validation. They demonstrate that we cannot directly use text samples related to categories in the target dataset as substitutes for test image samples, nor can we use the average performance of VLMs on open-source datasets as proxies for their performance on specific tasks. This problem statement is compelling and well-supported. 3.The method presented in the paper is novel and interesting. It leverages the category similarity between target tasks and open-source tasks to estimate the corresponding gap vectors and accuracy/ranking of VLMs on the target tasks, based on their gap vectors and accuracies/rankings on the open-source tasks. This idea is grounded in a reasonable assumption: the statistics of VLMs on similar categories are likely to be similar. By doing so, the method effectively utilizes the rich information contained in category similarity to make targeted use of the models' statistics on historical tasks. 4.The evaluation approach in the article is comprehensive and detailed. The benchmark includes 23 commonly used image classification datasets and 43 widely recognized VLMs (such as CLIP, CoCa, BLIP, BEiT, among others, with different pre-training methods and model architectures), making the experimental results highly convincing. Additionally, the article compares its method with various baseline approaches, including the current state-of-the-art method ModelGPT, and demonstrates its significant superiority over these methods. The article also conducts thorough ablation studies to validate the effectiveness of each component of the proposed method.

Weaknesses

1.The performance of ModelGPT is slightly inconsistent with the results in its paper. What are the reasons for this？ 2.The method of setting the \alpha value in formula 9 is not stated. Is the same \alpha used for all data sets? What is the exact value?

Questions

Refer to the weakness.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Response to Rebuttal

Thank the authors for the Rebuttal. I read the Rebuttal and other reviewers' comments. Since the authors have addressed my concerns, I would like to raise my score to boardline accept.

Impact of the new members in the model zoo

Another difference might stem from adding new members in the Model Zoo. Could you quantify the impact of this? For example, what is the performance of your approach using same Model Zoo in the LOVM paper?