Vocabulary-free Image Classification

Summary

This paper proposes the task of vocabulary-free image classification - image classification without a predefined set of categories (which is required in zero-shot classification). The paper further proposes the first method for this task - Category Search from External Databases (CaSED). For each given image, it first retrieves captions from an external text corpus, e.g. a large-scale caption dataset. The category names for the classification task are extracted from the retrieved captions via text parsing and filtering. For the classification, a scores fusion is conducted between 1) image-to-text matching score between the input image and the category names, and 2) the text-to-text matching score between the average embedding of retrieved captions and the embeddings of category names. The technical contributions are 1) The paper proposes the task of vocabulary-free image (VIC), to overcome the constraint of requirement of a predefined set of class names 2) The paper proposes an approach for VIC task via extracting category names from an external large-scale text corpus

Strengths

1) The paper is clearly written and easy to follow 2) Vocabulary-free image classification is an interesting task as it requires no prior knowledge of semantic class space of visual inputs.

Weaknesses

1) Some details of baselines and evaluation metric are missing. See questions. 2) The choice of the caption database (as the text corpus) might have a big impact on the performance. Guidances / criteria of selecting a good text database are missing. 3) A baseline is missing: WordNet and English words contain different semantic words than words covered by captions in the subset of PMD. It would be interesting to extract the words from the captions in the subset PMD, and apply CLIP for image-text matching. 4) In Table 4(b) using YFCC100M (29.9M) as the text corpus leads to worse results than the case of CC12M (10.3M) and Redcaps (7.9M). Using CC12M alone leads to better results than using all the five datasets (54.8M). Further explanations are required. It would be interesting to report, besides the number of caption, a) the number of objects / nouns covered in each database, and b) the relevance of captions in a database to the image dataset, e.g. by reporting the semantic similarities between captions and semantic labels of the image dataset, semantic similarities between captions and the visual contents of the image dataset. Minor: 1) Table 4(b): asterisk meaning missing (I assume it means a subset of YFCC100M)

Questions

1) In Sec 5.2 line 275, the paper mentions that “applying CLIP on top of a pre-defined large vocabulary is not effective” based on the inferior performance on WordNet (117k words) and the English Words (234k words). However, as the first step, CaSED uses CLIP to retrieve K closest captions to an input image from a large-scale text corpus (54.8M captions). Why is it effective here? 2) For the cluster accuracy (line 242), is it necessary to make sure that the number of predicted labels (number of clusters) should be equal to the number of ground truth labels? How to make sure? What happens if these two are not equal? 3) For the baselines of BLIP-2 captioning (line 253), each image has one description. How to compute the semantic relevance and the cluster accuracy? How to determine the predicted label from this description? Is the description directly treated as a predicted label?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

One missing limitation is that the method requires selection of a text database that is semantically relevant to the contents of the query image dataset. The choice of text corpus might have a big impact on the performance.

Summary

This work introduces Vocabulary-free Image Classification (VIC), a task that aims to assign an image to a class in a large, evolving semantic space without a known vocabulary. The proposed method, Category Search from External Databases (CaSED), utilizes a vision-language model and an external database to extract candidate categories and assign the best matching category to the image. Experimental results demonstrate the superiority of CaSED over other frameworks, offering efficiency and promising future research opportunities.

Strengths

The strengths of this work can be summarized as follows: 1. Exploration of Vocabulary-free Image Classification (VIC) as a task, which overcomes the limitations of existing Vision-Language Models (VLM) for image classification. 2. Proposal of CaSED, a training-free method for VIC that utilizes large captioning databases. CaSED does not require additional parameter tuning or fine-tuning of textual and visual encoders. 3. Consistent outperformance of CaSED over a more complex VLM, BLIP-2, on VIC. CaSED achieves superior performance while utilizing significantly fewer parameters. 4. Introduction of specific evaluation metrics for VIC, providing a valuable reference for future research and benchmarking in this domain.

Weaknesses

1. The most interesting part of this paper is the new task: vocabulary-free image classification. However, it is already studied in the existing work: https://openreview.net/forum?id=sQ0TzsZTUn [1]. The major differences between this paper and the existing work are as follows: a) The authors of the current paper extend the evaluation by using a larger vocabulary set -- BabelNet, compared to the existing work. b) The authors adapt the text-to-text score to find the best matching class. From the perspective of the reviewer, these two differences may not be considered major advancements. Further analysis and comparison with the existing work are required to fully understand the extent of novelty and contributions of the current paper. 2. To enhance the clarity and completeness of the paper, it would be beneficial for the authors to provide additional information on how to calculate the classical Cluster Accuracy, particularly considering the open vocabulary nature of the predicted labels. Given that the vocabulary is unconstrained, it is important to explain how the accuracy metric accounts for potential variations in the predicted labels. 3. It is worth considering that most of the class names in vocabulary-free image classification may consist solely of nouns. Consequently, retrieving information from image captions, which typically contain more diverse linguistic elements, could be inefficient and computationally expensive. This raises a valid concern about the practicality of such an approach, as it may not yield meaningful results and could potentially waste computational resources. The authors should address this issue by discussing alternative strategies or potential optimizations to mitigate the computational burden associated with retrieving information from image captions. [1] Han, Kai, et al. "What's in a Name? Beyond Class Indices for Image Recognition." arXiv preprint arXiv:2304.02364 (2023).

Questions

Please address my comments in the weakness section.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

Current approaches to zero shot classification are limited by the assumption of an existing set of candidate classes. This paper presents a novel task, Vocabulary-free Image Classification, where semantic categories need to be automatically mined. This is important when generalising to a new domain, or the domain evolves over time. To address this task, the paper proposes to search for categories from an external database. While conceptually simple, this proves to be an effective approach, and performs better than using captioning models or large lexical databases like WordNet. The paper also proposes metrics to evaluate on this new problem.

Strengths

- The paper addresses an important limitation of prior zero-shot classification approaches. - The importance of the proposed task, the motivation behind the method, and the method itself, are clearly laid out. - The components of the method are well ablated - Multiple baselines are proposed and analysed - It presents an avenue for future research on how to discover semantic concepts in the V+L space

Weaknesses

1. Text-to-text score. Using CLIP as a scoring function for image-to-text makes sense. However, it is not obvious that a CLIP text encoder is optimal for the text-to-text score. L207-208 say that the CLIP text encoder ”focuses on the visual elements of the caption”. This could be backed by showing inferior performance when using a text-only model for the text-to-text score. Furthermore, an addition to the semantic space representation experiments (L136-170) would be a model that uses CLIP to retrieve captions, and then a text-only encoder to match the captions to a semantic class. 2. Reproducibility. The paper does not mention how BLIP was prompted to generate class names 3. Table 4a. The numbers on the last row of table 4a (retrieval with vis + lang scoring) correspond to the avg. values on Tables 1, 2 and 3. However, the first row (BLIP2) does not correspond to any row on these tables. (Related to 2. Reproducibility -- it not clear how the different settings differ) 4. Multimodal scoring results. Table 4 a) shows that scoring with the text-to-text score performs better than image-to-text score, and adding the image-to-text contributes a small amount. This means that the average caption is the most informative signal. That makes the statement on L192-193 "The visual information is the most reliable source for scoring the candidate categories" invalid. It is indeed an interesting finding and some intuition behind that would be useful. It would be interesting to see the performance of an ever simpler baseline that only uses text - select the class that has the highest number of occurrences in the retrieved captions. 5. Candidate classes. What is the distribution of the number of extracted candidate classes? And how does it differ as you change the number of retrieved captions? 6. Generalization: One of the motivations behind this work is the applicability of ZS classifiers to domains that might not have a closed set of classes available. However, the current approach relies entirely on datasets of image-text pairs, from which captions are mined. For a domain that is not sufficiently represented in such datasets (e.g. medical images), the proposed method will fail. A discussion of how text-only databases can be used to handle that would be nice to see. 7. Database. In Table 4b you show that simply using the CC12M set of captions performs better than the contracted one (Ours). Why not use that?

Questions

1. Cluster Accuracy. When measuring cluster accuracy, one can imagine that in the vocabulary free case, there might be many more predicted labels that in the ground truth (up to the number of samples of the dataset). How is that handled? + Please see weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Then limitations look good. A limitation not discussed is Weaknesses 6. Generalization

Summary

The paper presents a method to perform image classification by using the similarity text and image embeddings of a large vision-language model (VLM). The method is called "vocabulary-free" because the large vision-language model doesn't have a particular set of classes (vocabulary) in mind, and (contrary to zero-shot approaches), the user does not need to provide a list of prompts. In particular, a set of k closest captions to the image embedding is retrieved from the VLM, and this candidate set is later scored to predict the image class. The underlying VLM is a ViT-L CLIP model.

Strengths

- The paper shows very good results when compared with different reasonable baselines (see Table 1, in particular). The experiments use several popular image classification datasets, and use different metrics, and the proposed method is significantly better than the baselines in virtually all cases, and certainly on average. - The figures in the paper help understanding the proposed approach (Figure 3) in detail and the difference between "vocabulary-free" and zero-shot image classification. - The paper contains results from ablation studies used to justify certain design choices (e.g. number of candidate captions retrieved from the database), the importance of the visual and language scores, etc (see section 5.3). I found these experiments very interesting (modulo some questions that were unanswered, see later comments).

Weaknesses

- The term "vocabulary-free" is a bit misleading, since the VLM does have a vocabulary built-in, i.e. the so-called "semantic space". It's impossible to classify an image representing an object outside this semantic space. However, it's true that the semantice space is far larger than the class space of any of the datasets used for evaluation (although not necessarily a superset). What the authors mean by "vocabulary-free" is precisely defined in the paper (section 3), so this weakness is not critical, but as mentioned before, I find the term not accurate. - My main concern with the paper is regarding the use of the datasets used to train the VLM model used by the proposed method. Section 5.1 mentions that the nearest-neighbours are searched over a collection of 5 large datasets, and the authors ablate the impact of each dataset in section 5.3 (table 4b). However, it's unclear the impact of the dataset used to train the VLM when compared with the baselines. To be more specific: is the proposed CaSED method than BLIP-2 VQA because the dataset used to perform knn is bigger, because the embeddings are better, or because knn+scoring is better than VQA (regardless of the dataset)? - Lines 164-165 refer to the number of parameters as a reason for better/worse speed. This is technically wrong. The number of parameters can be (extremely highly) correlated with the speed, but it's not a direct cause of it necessarily. Thus, if arguments about speed are to be made, real runtime/query (on a given hardware) or -at the very least- FLOP/query (amortized over a given batch size) should be given instead. --------- Update after rebuttal: The authors have (quite successfully) addressed my comments in their rebuttal. Thus, I'm slightly increasing my score.

Questions

- Have you tried with other strategies to compute the final candidate scores? For instance, why not a log-linear average between the text-to-text and image-to-text scores?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have adequately adressed the limitations and broader impact of the paper in section 6.

Rating changed into borderline accept.

I appreciate the efforts of the authors and feel that most of issues are addressed (more or less). However, I am still concerned about the limitation of selection of a proper text corpus which has a big impact on the performance, as there is no clear unsupervised criterion guiding in this regard. Overall, I change my rating into borderline accept.

Thanks for the response and still concern about the novelty.

I appreciate the response on my concerns. The difference with the other paper is clearer to me now. First, I agree that compared with the paper I pointed out, this submission leverages an external text caption database, while the other work uses an unconstrained dictionary. This is slightly different. Second, the statement that “our approach generates the vocabulary independently among samples, working in an online fashion” is not true. The CaSED method is not generating any vocabulary independently. Instead, a predefined and fixed large-scale text database is given, and for each image, the large-scale text database needs to be fully scanned to retrieve the relevant captions. Hence, no independent vocabulary generation is involved. It is retrieval from the extra text database. After reading all reviewers comments and the rebuttal response, I remain rather concerned about this submission. 1. The novelty of the paper is rather limited. a. From the method perspective, the use of VLM for classification is not new, leaving the main pitch of this paper to be the use of an external text database to retrieve the most “relevant” captions, and then assign a category name based on the retrieved captions. This is a minor change on top of the CLIP model. Meanwhile, using external text database to retrieve helpful information is a very common method in NLP, e.g.[a], as also mentioned in the related work. However, this idea has also been well studied in the computer vision literature, aside from those mentioned in the submission, there are other ones that are more relevant, e.g., using an external text database to facilitate image recognition [b], image captioning [c], generalized class discovery [d], among others. [a] Lewis,Patrick, et al, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020 [b] Choudhury, Subhabrata, et al, The Curious Layperson: Fine-Grained Image Recognition without Expert Labels, BMVC 2021 [c] Ramos, Rita, et al, Retrieval-augmented Image Captioning, EACL 2023 [d] Ouldnoughi, Rabah, CLIP-GCD: Simple Language Guided Generalized Category Discovery, arXiv:2305.10420   b. From the task perspective, the so-called “Vocabulary-free Image Classification” (VIC) is not “vocabulary-free”, as pointed by other fellow reviewers, due to the fact that the very large vocabulary, in the form of a large-scale caption dataset, is required. It says the “The objective of VIC is to assign an image to a class that belongs to an unconstrained language-induced semantic space at test time, without a vocabulary”. This is not true. The used caption text dataset is a superset of the vocabulary, which contains more abundant information that simply an open-vocabulary.   Indeed, it appears to me that the VIC is essentially the open-vocabulary recognition, for which there are plenty of studies already, with methods addressing even harder problems of object detection (i.e., box level image recognition) and segmentation context (i.e., pixel level recognition). How does VIC differ from them? The whole literature along this direction was omitted.   For example, Ghiasi, Golnaz, et al, Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV 2022 Zareian, Alireza, et al, Open-Vocabulary Object Detection Using Captions, CVPR 2021   2. The choice of the text database is very crucial. There is no guarantee that the text database can cover the semantic names of the testing images. This can not be reliably used in practice. On the other hand, if the semantic names are known beforehand, the problem becomes a close-set classification problem, contradicting its motivation. In this sense, there is no guarantee that the method can work or not in praticse.   3. The inference time is as slow as the generative methods, e.g., BLIP-2, which is handling a more difficult problem, by directly generating the caption containing the name for the given images. For the comparison with BLIP-2, which was questioned by other reviewers as well, I also think it may not be fair. The proposed method is given a database to “look up” the category names, but the BLIP-2 merely leverages the input image to directly predict the category names. Therefore, the novelty is deeply disconcerting from both the standpoint of the task at hand and the methodology employed. In light of the comprehensive examination of comments and discussions, my apprehensions have been significantly heightened. Hence,I would keep my initial rating.