Summary
In this paper, the authors argue that the visual features learned by the vision encoder of the CLIP model, trained using contrastive learning, might be suboptimal for image-to-image matching tasks. The CLIP model is primarily trained for cross-modal matching rather than intra-modal matching.
The authors propose that instead of directly performing intra-modal matching among images in a few-shot scenario, it's better to initially identify which objects in the text are similar to the query image, as well as which objects the few-shot training images resemble in text. The fundamental idea is that if two objects share a similar set of both similar and dissimilar items, the two images are likely similar to each other. This strategy avoids directly computing the intra-modal distance, which the authors argue is suboptimal, and instead relies on cross-modal distances, for which the model was trained. Thus, the authors propose an approach for representing an image in terms of its similarities, termed "CODER."
However, the authors note that to create a robust representation, a dense sampling of relevant information about other categories is required, meaning a good feature vector representation. To achieve this, they employ a language model, referred to as an "Auto Prompt Generator," to generate a variety of prompts. These prompts help measure the distance with the image, thereby enhancing the CODER's representation beyond solely using class names.
Specifically, the Auto Prompt Generator aims to generate different textual prompts that a given image can be matched to, to form the CODER feature vector. Their approach involves creating five types of prompts: category name-based prompts, attribute-based prompts, analogy-based prompts, synonym-based prompts, and one-versus-one-based prompts. The first two were previously explored in prior work. To find analogous classes, they prompt GPT by inquiring about other categories similar to a given class and create prompts from those various classes listed. For synonym-based prompts, they use WordNet. As for one-versus-one texts, they ask about the different visual features between different classes in a photo. These prompts are then used to construct a feature representation.
For zero-shot classification, the authors adopt a heuristic approach. They first identify the most similar category name and synonym name. Next, they find the mean of maximum elements corresponding to the attributes and synonyms. This process generates a similarity score for a given image and class. Finally, they perform a re-ranking phase using the one-versus-one-based prompts on the top two predicted classes using their CODER representation.
In few-shot classification, the authors build on TIP-Adapter by incorporating their new CODER features. After integrating their features, they follow the original approach, calculating an affinity score between a test image's CODER and the support set images. This score forms a weighted sum of labels from the support set, ultimately used to make a class prediction.
The authors conducted experimental comparisons of their approach on zero-shot and few-shot classification tasks. For zero-shot, they compared it against vanilla CLIP and VCD. The proposed approach demonstrates improvements in performance across many settings compared to existing methods, except in one particular dataset where it exhibits a significant drop. Similar results are observed in the few-shot settings.
Strengths
Before delving into the technical strengths of the method, it's important to note that the paper itself is well-written, clear, and easy to understand and follow.
Regarding the approach itself, the underlying intuition makes sense. The authors assert that a crucial insight, based on K Nearest Neighbors papers, emphasizes the importance of having a dense sampling of neighbors to achieve good performance. They argue that existing approaches, like VCD, lack this essential aspect. Their primary idea revolves around how to create more neighbors. Another key insight is that measuring distances intra-modally between images, which is not within CLIP's training objectives. Therefore, rather than calculating distances across images, the authors suggest comparing the distances between an image and text with the distances between another image and text. They claim that this method is a superior way to measure similarity compared to directly computing feature distances.
The concept itself is straightforward and likely involves minimal computational overhead, simply entailing a dot product between pre-computed prompt templates. Moreover, the authors extend existing approaches, like VCD, in an intriguing manner. For the zero-shot case, the authors propose further refinement after performing similarity measurements in cross-modal distances. They suggest examining a set of prompts that specifically focus on the top two classes. This step, concentrating on the distinguishing features of those two classes, is particularly intriguing. The initial stage could be seen as a coarse retrieval step, while the subsequent stage emphasizes differentiating the fine-grained features between different objects.
Experimentally, the authors demonstrate significant results for zero-shot and few-shot classification. In zero-shot classification, they observe a consistent improvement of 1 to 3% across various model sizes they tested, with increases of up to 7% on some datasets.
Weaknesses
In my view, the paper has several significant weaknesses. Firstly, I'd like to highlight the similarity of the proposed approach with the VCD methodology. In VCD, authors perform classification by initially extracting a set of prompts from a language model and then classifying the object based on a weighted sum of the similarity of these prompts. As the authors note, the first two prompt styles have been explored in previous works. However, these earlier papers, such as VCD, conducted a more comprehensive analysis. For example, VCD addressed bias in prediction through their approach. Hence, it seems the authors are attempting a similar method to VCD but by adding extra prompts from the language model. This, on its own, doesn't present a significant technical advancement. Although the authors introduced prompts to capture analogous images and others as mentioned, it appears more of an incremental approach than a groundbreaking innovation. Among these prompt types, the one versus one prompt seems somewhat more innovative. Here, after identifying the top two possible classes, the authors use their one versus one prompt to determine the superior one. However, again, this remains of minor novelty.
Regarding the few-shot approach, the author mainly makes use of TIP-Adapter by merely replacing their features with CODER features. This doesn't seem to contribute particularly novel ideas either.
Additionally, while the authors included experimental analyses for few-shot and zero-shot classification, the depth of analysis in the paper seems somewhat limited. Compared to the VCD paper, which provided a detailed analysis of the technique along with various extensions and qualitative results, the analysis in the proposed approach seems lacking.
The observation the authors make about CLIP features not being well suited for intramodal similarity is valid. However, the solution the authors seem to propose doesn't entirely address the underlying problem. Wouldn't a better solution be to enhance CLIP's features to better handle intramodal similarity? This might be a more direct approach. For instance, enforcing a self-supervised objective during training, such as requiring an image to be close to the representation of its augmented image. Several such self-supervised objectives exist. This raises the question of whether off-the-shelf CLIP features are genuinely the best fit for this task, as opposed to a model specifically trained to capture intramodal semantic locality. Thus, the approach seems to rely on a model that might not be ideally suited for the task without either selecting a better model or trying to enhance the model's learning. It's not evident that this approach is superior to simply training a CLIP model that enforces intramodality constraints.
The key weakness, of course, is that the authors' proposed approach initially requires using a language model such as Chat GPT to extract a large amount of knowledge for the given classes. This might be challenging for individuals without access to Chat GPT APIs. Referring to this as a "training-free" approach might be misleading because although it doesn't involve training a classification model in the traditional sense, it does require Chat GPT for diverse knowledge, essentially creating a sort of prompt-based model. It's not training in the conventional sense of weight updates, but rather creating a set of prompts that act as weights. Hence, labeling this a training-free approach seems inappropriate, as it initially requires interaction with a language model to construct a knowledge landscape.
Again, however, the most critical issue with the paper appears to be the lack of significant technical innovation. The primary contribution seems to be the addition of new types of prompts compared to the VCD paper. Therefore, the contribution appears relatively limited.
Questions
1 - rather than the CODER features used, what is the difference between TIP-adapted and authors' approach?
2 - what about very fine-grained tasks where the language model does not have good knowledge about the particular type of image? In this case, image-to-image similarity might be required. I am thinking of images of types of cells, very particular types of medical instruments, etc. In this case, the proposed approach seems weak.
3. Do the authors' believe it is consistent to call this a "training free approach" when ChatGPT is required to create a sense of prompts which function as the weights of a classifier?
4. Why not fix the underlying problem and use a different model (large-scale) or train/fine-tune clip with an intramodality loss? Seems that is the more elegant fix.
Rating
5: marginally below the acceptance threshold
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.