ChatGPT-Powered Hierarchical Comparisons for Image Classification

Summary

The article presents an approach for refining the classification performance of a vision-language model (i.e. CLIP), using descriptions derived from a large-language model (i.e. GPT-4). Different from previous attempts in this direction [26], directly extracting a discriminative description for each of the classes, this work proposes a hierarchical approach where more specific class descriptions are available at each layer. Starting from the class descriptions of [26], the text embeddings are used to perform clustering, creating groups of similar classes. Then, the LLM is used to obtain further descriptions to discriminate classes within each similar group. This process can be repeated iteratively until the number of classes in a group is low. Experiments on various benchmarks show that the proposed model consistently outperforms CLIP and the direct descriptions of [26].

Strengths

- The idea of splitting classification into multiple stages by asking for increasingly more specific descriptions is interesting as it allows the model to focus on easy separation first and increasingly more fine-grained ones later in the process. It also inherently gives interpretability, as we can check which type of descriptions lead to high scores for a certain class (Figure 4). - Section 3.2 describe the type of query used, while the supplementary (Section 1.1) shows examples of failure cases of naive querying strategies. - Despite the scores being very close among compared classes, Figure 4 provides a clear view of the type of descriptions involved in the classification process. Similarly, Figure 6 (despite being hard to interpret without the caption) shows the step-by-step changes in the model predictions, further complementing the info in Figure 5 on the hierarchical process. - The framework is flexible, as it can be readily applied to new versions of large-language models and vision-language ones.

Weaknesses

1. Despite consistently outperforming [26], the gap between the two models is relatively low: i.e. in Table 2, the average gain is always below 1% in accuracy. This comes unexpectedly as the model is initialized using descriptions extracted as in [26] (lines 134-136), and the article stresses the advantages of the proposed approach (e.g. captions of Fig. 1, lines 75-77, lines 139-142, lines 171-174). Moreover, the proposed approach increases the computational cost, by increasing the number of text-image comparisons (i.e. the summation in Eq. 5). The claims on the performance/advantages should be revised to avoid overstating the contribution, and the computational cost discussed/reported. 2. The article reports limited analyses w.r.t. the method's hyperparameters. For instance, the model uses a factor $\lambda$ to balance the default CLIP prediction with the obtained hierarchical one (Eq. 6) but the impact of $\lambda$ is not analyzed. Similarly, this applies to the number of clusters $N$ and the size of the leaf nodes for the stopping criterion, $l$. It would also be interesting to show how the performance varies depending on the depth in the hierarchy, to confirm that deeper predictions lead to better performance. 5. $\tau$ is defined as the tolerance score to mitigate the above issue (Eq. 5, line 194). However, this value is set to 0 in the experiments (line 201). If $\tau$ is not used, it should not be defined, otherwise, it is important to specify the cases where it is used and analyze its impact. 3. [26] tests also on the EuroSAT dataset of satellite images, where providing natural language descriptions is challenging. This dataset is not included in the current experiments, but would further show the potential advantages of the approach in difficult classification settings. 4. Minor point: lines 185-194 describe the need to avoid that unrelated objects receive high scores for very specific descriptions. A strategy to avoid potential differences could be to just perform classification group-wise at each level of the hierarchy. This would work by "filtering" the classes by excluding every class that is not part of the predicted group for the next level of the hierarchy. Would this strategy be feasible? and in general, is there a reason for using the average (Eq. 8) rather than other aggregation mechanisms of the hierarchical scores?

Questions

Following on the previous points: 1. Why is the gap between [26] and the proposed approach low? 2. What is the computational cost of the proposed model? 3. What is the impact of $\lambda$, and $N$? 4. How does performance vary w.r.t. the depth of the prediction? 6. Is $\tau$ used? If so, how does the performance vary? 5. What is the motivation behind using the average as the aggregator in Eq. 8? Please, while answering, clarify also how the potential additional analyses/discussions would be included in the revised manuscript.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Section 5 discusses various limitations of the model, highlighting a fundamental one, i.e. the assumption that a category is easy to describe via natural language. Overall, I find the assessment of the limitations fair and thorough.

Summary

This paper presents a hierarchical comparison method for zero-shot open-vocabulary image classification. By leveraging CLIP and Large Language Models (ChatGPT here), the authors build class hierarchies by recursively comparing and grouping classes. This hierarchy enables image classification by comparing image and text embeddings at different levels, improving accuracy and interpretability. The proposed method outperforms existing methods, achieving state-of-the-art performance on various datasets.

Strengths

The proposed method exhibits high interpretability, as it draws inspiration from how human beings hierarchically organize visual and linguistic concepts in the real world. This intuitive approach enhances the understanding and explainability of the image classification process. The efficient utilization of the extensive pretraining knowledge from ChatGPT/GPT-4 significantly improves the quality and granularity of the text descriptions. The evaluations are conducted on 6 datasets and 4 different backbone settings, ensuring a robust assessment of the framework's effectiveness and generalizability.

Weaknesses

It is worth considering recent open-source LLMs like LLaMA, Alpaca, and Vicuna for generating descriptions and conducting comparisons. These LLMs could provide alternative options and insights for the generation and evaluation of descriptions in the proposed framework. The evaluation in the paper only includes one baseline method [26], which may limit the assessment of the proposed approach. All the datasets used in the evaluation are in the general domain, which might not fully demonstrate the generalizability of the proposed method across various domains. Including datasets from diverse domains would further validate the effectiveness and robustness of the approach.

Questions

Please see the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The utilization of ChatGPT/GPT-4 for generating descriptions incurs a cost, which could potentially restrict the reproducibility and impact of this work.

Summary

This paper shows an example of introducing the power of LLMs in the zero-shot classification. This paper proposes a framework (or a process flow) rather than proposing a new mathematical algorithm. However, as shown in the experimental results section, it is demonstrated that the classification accuracy is about 1% point better than the state-of-the-art.

Strengths

+ It is impressive to see how the integration of grouping and Language Model-generated descriptions to build knowledge trees can boost zero-shot classification performance. This is a compelling application of Large Language Models.

Weaknesses

- I am afraid that the classification accuracy would depend on the performance of LLMs, but the detailed discussion on the accuracy of LLMs and its relation to the final classification performance is missing. It would also be interesting to compare multiple LLM models. - I admit that the classification performance can be improved as can be seen in Tables 1 and 2. However, only the numbers are compared in those tables. The authors might want to discuss under what circumstances the proposed LLM-based one works better than the baselines and under what circumstances it does not. In other words, please discuss the remaining problems or limitations of the proposed method. This is just a comment to improve the paper. The texts in figures are too small, especially in Table 4. The authors might want to make them the same size as the text in the main body.

Questions

Please see “Wekness” part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Please see “Wekness” part.

Summary

This paper argues that CLIP is limited due to the semantic ambiguity of similar classes. To overcome this, the paper proposes to construct a class hierarchy with large-language models and introduces a novel image classification framework based on hierarchical comparisons that compare embeddings at each level. Extensive experiments show promising results on multiple datasets under the zero-shot image classification task.

Strengths

-This paper is well motivated. Using a class hierarchy to disambiguate the semantic embeddings of similar classes make a lot of sense. -A novel hierarchical classification framework is proposed. To my knowledge, constructing a class hierarchy from LLMs is new. It is interesting to see this new application of LLMs. I also like the proposed plug-and-play hierarchical classification method which does not need to retrain the CLIP model. -Empirically, it consistently improves the CLIP and Open-CLIP on 6 datasets under the zero-shot image classification task. The interpretability is another interesting property of the method.

Weaknesses

-Comparisons between the class hierarchy generated by LLMs and the ones labeled by humans e.g., WordNet are missing. -What is the impact of the hyperparameters, or example, the threshold to construct the hierarchy?

Questions

I do not have additional questions at this stage.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations are discussed in the paper.

Thank you for the comments.

All of my concerns have been solved. I still keep on the acceptance side.

Thank you and further clarifications

I thank the authors for the thorough reply, answering the questions raised in all reviews. I especially appreciate the details on the computational cost and the provided analyses on the hyperparameters. For completeness, I would like to ask further questions/clarifications: 1. According to Fig. 3, the depth of a class varies based on the clustering algorithm (please correct me if I am wrong). Given that various classes are found at different depth levels, how have the results above been computed? i.e. does depth N mean that all classifications for happen at depth N of the tree but if a leaf is found at node M<N, its value at M is used? 3. While the depth analysis shows a large improvement from 1 to 2 levels of the hierarchy, the description obtain by [26] achieve already a gain comparable to the depth 6 results (-0.4%) according to Table 2. This shows that asking for contrastive descriptions is less effective than asking for detailed ones if done in just a few rounds. Is this "issue" linked to the type of prompt used or to the clustering itself that gathers hard-to-contrast groups in the first levels? 2. The depth of the tree is proportional to the choice of the threshold value $l$. Will an explicit study on $l$ be included? 4. Are there failure cases in which an increase of depth in the decision leads to wrong predictions? If so it would be interesting to discuss them and why they happen. 2. Since $\tau$ is not used/does not have a positive impact on the final average results (according to Table D), will it still be included in the manuscript?