Weaknesses
* the presentation of the method in Section 3 is quite hard to follow:
- first, it starts by "CLIP [65] formulates image classification as a retrieval task by using the textual class labels (...) and encoding them with the language encoder." which is a very imprecise way of putting things. CLIP does not "formulate" any task, it is a neural model that learns visual representation with natural language supervision. It can be instructed in natural language to perform image classification but it requires a specific approach (textual prompt) in each case.
- the notations on lines 107-114 may be clearer. For example, it starts by considering an image $I$, then describes CLIP by splitting it between an encoder and a final linear layer that are no longer used in the following, to finally considering the similarity $s(i,j)$ between an image (that was $I$ at the beginning) and a text (never introduced)
- lines 115-121, the text is unclear. The sentence "a unified set of $\mathcal{D}$ class-agnostic textual descriptors of length D" seems to mean that there are $\mathcal{D}$ 'descriptors' and that each is of length D. However, the following leads to doubt about that since the manuscript reports "D = 4,229 after discarding repetitive descriptors across the entire pool" suggesting that D is the number of 'descriptors'. Moreover, on line 118 "Concepts in $\mathcal{D}$" suggests that $\mathcal{D}$ is now the name of the set and not the number of concepts. After several readings, I think that the authors refer to the "length of the set" as Python programmer while one should consider the *size* of the set, meaning that D=4,229 is the number of elements in the set $\mathcal{D}$.
- it is nevertheless still unclear what is a 'descriptor'. On line 120, an example of 'textual descriptor' is given ("can be hung from a tree") suggesting that it is actually a *description*. However, it is unclear how these descriptions are obtained, the only hint being on line 115: "We utilize an LLM to generate descriptors...". How these LLMs are used? From which data (what is the prompt)? While lines 102-114 introduce many notations just before, the names of the classes (one guesses $\mathcal{Y}$ on line 103) are no longer used in the following
- on line 134, it is not clear what is "the zero-shot predicted class" nor how it is obtained. After some consideration on mutual information (lines 122-126) the overview of the method starts with a "given set of images" (line 127) thus it is unclear to which image (images? patches?) this "zero-shot predicted class" relates to.
- In the same vein, the (same?) "zero-shot prediction of CLIP" at the beginning of section 3.2 should relate to an image but it is not clear which one
- Line 205, how the "importance to the image" is determined?
- Last but not least, it may be worth explaining how the "explanation" is provided by the proposed model once the "mutual information dynamics" is obtained
* Similarly, Section 4 starts by explaining that an evaluation was conducted but the reader does not know for which task nor on which dataset; ImageNet is used for training (line 222) thus one guesses that it deals with zero-shot image classification (cf. title of the article) on the validation set of IM-1000? However, no implementation is provided for the experiment which results are reported in Table 1: which visual encoder is used (several are considered in the following)? how are they trained? Are the Vit/CNN pre-traind on some data? Which textual encoder is used? what is the LLM used to get the textual description of the classes and how is it done in practice? Moreover:
- the manuscript refers to [7] to explain the metrics "Accuracy Drop" and "Accuracy Increase" (lines 224-225). However, in [7] the word "Increase" does not appear and the only occurrence of the word "Drop" does not refer to a metric. It is only in Appendix F that the reader can understand that the name of the metrics was changed. Moreover, if the "adaptation" of the metric is explained (still in the appendix) for the proposed method, it is not clear whether such an adaptation is required for the other methods (and whether it is the same as for the proposed approach). Previous works such as [37, 38] report the accuracy of each model, which seems relevant.
* Previous works such as [18, 37, 38] usually evaluate their approach on several datasets. However, one must admit that they do not report quantitative evaluation in terms of explainibility.
* On the opposite, it would have been relevant to compare the approach to those that specifically deal with XAI such as [a,b,c], in the vein of [d] which is a reference that could be cited. More generally the proposed work dealing with prototypes, it could be relevant to cite some of the works that adopted such scheme since [e].
[a] Wang, C., Liu, Y., Chen, Y., Liu, F., Tian, Y., McCarthy, D. J., Frazer, H., and Carneiro, G. (2023). Learning support and trivial prototypes for interpretable image classification. ICCV
[b] Lei, Y., Li, Z., Li, Y., Zhang, J., and Shan, H. (2023). Lico: Explainable models with language image consistency. NeurIPS
Information Processing Systems
[c] Wan et al (2024) Interpretable Object Recognition by Semantic Prototype Analysis, WACV
[d] Chen et al (2019) This Looks Like That: Deep Learning for Interpretable Image Recognition. NeurIPS
[e] Saumya Jetley, Bernardino Romera-Paredes, Sadeep Jayasumana, and Philip Torr (2015) Prototypical Priors: From Improving Classification to Zero-Shot Learning. BMVC
**minor points**
- the text in Figure 1 is barely legible. On a standard A4 page, it is hard to read the list of concepts above (d).
- reference [21,52] are incomplete
- reference [29] is wrong, it has been published in NIPS 2012. Actually, a 2017 paper was published in Communication of the ACM, but it is clearly written (bottom right) that the original paper was published at NIPS 2012.
- the date is missing in [30]
- [49] both refer to the article and the (arxiv) preprint
- line 224: low --> lower
- The derivation in Appendix C is not very useful as the result is well-known for uniform distributions. However, a discussion on such uniformity may nevertheless be relevant.