In this paper, we present Language Model as Visual Explainer LVX, a systematic approach for interpreting the internal workings of vision models using a tree-structured linguistic explanation, without the need for model training. Central to our strategy is the collaboration between vision models and LLM to craft explanations. On one hand, the LLM is harnessed to delineate hierarchical visual attributes, while concurrently, a text-to-image API retrieves images that are most aligned with these textual concepts. By mapping the collected texts and images to the vision model's embedding space, we construct a hierarchy-structured visual embedding tree. This tree is dynamically pruned and grown by querying the LLM using language templates, tailoring the explanation to the model. Such a scheme allows us to seamlessly incorporate new attributes while eliminating undesired concepts based on the model's representations. When applied to testing samples, our method provides human-understandable explanations in the form of attribute-laden trees. Beyond explanation, we retrained the vision model by calibrating it on the generated concept hierarchy, allowing the model to incorporate the refined knowledge of visual attributes. To access the effectiveness of our approach, we introduce new benchmarks and conduct rigorous evaluations, demonstrating its plausibility, faithfulness, and stability.
Paper
Similar papers
Peer review
Summary
This paper builds an attribute tree using LLMs to explain image classifiers. To this end, the paper designs a framework that expands and prunes nodes for the tree. Using this framework, the paper provides a richly annotated version of CIFAR and ImageNet, which is 5 times more expressive than prior works. With this attribute tree, the paper outperforms previous decision tree approaches such as NBDT.
Strengths
- New datasets with rich attributes could be valuable for the community, though the necessity of such large attribute sets should be further justified (W3). - Using rich attributes, the paper outperforms previous NN-based decision trees, such as NBDT. - The paper is quite dense, containing a lot of information.
Weaknesses
1. Explanation is more than an attribute tree The paper focuses on building attribute trees using LLMs. However, there are more diverse ways to interpret the behaviors of vision models, particularly using language. For example, [1,2] use vision-language models to understand the failures of vision models through language explanations. [3-5] use language to derive concepts for concept bottleneck models (CBMs), though this paper only cites [5]. [6] generates language to explain the neurons in the network. Among these approaches, the paper should discuss the advantages and drawbacks of the proposed method of creating an attribute tree. For this reason, I believe the current paper title is overstated. I recommend clarifying its scope in the title to focus on the construction of hierarchical attributes and decision trees. [1] Kim et al. Discovering and Mitigating Visual Biases through Keyword Explanation. CVPR 2024.\ [2] Wiles et al. Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning. arXiv 2023.\ [3] Yuksekgonul et al. Post-hoc Concept Bottleneck Models. ICLR 2023.\ [4] Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR 2023.\ [5] Yang et al. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023.\ [6] Hernandez et al. Natural Language Descriptions of Deep Visual Features. ICLR 2022. --- 2. Insufficient qualitative insights The paper mostly focuses on quantitative metrics such as plausibility, faithfulness, and stability, which support the validity of the proposed explanation. However, an explanation is only meaningful when it provides new insights to humans. In this sense, the paper should carefully examine individual data points and their corresponding explanations and discuss the lessons learned from them. I know that Figure 6 provides some examples of decision trees. However, the paper could delve deeper, for example, by addressing: 1) What are the common issues with current classifiers? 2) What are the differences between different models, such as ResNet vs. ViT? As the paper has built a tool, there are numerous directions to explore using it. --- 3. Necessity of enormous attributes The paper creates datasets with rich attributes (H-XX), which are 5 times more expressive than the prior datasets (DR-XX). However, do we really need all these attributes? More analysis and justification of the attributes should be provided, clearly showing the limitations of prior works.
Questions
Can the method be improved using multimodal models like GPT-4V instead of language-only models like GPT-4?
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Discussed in the Appendix.
Response to the Rebuttal
Thank you for the detailed response and experiments. I believe the construction of hierarchical concepts is valuable for XAI, despite concerns about its usefulness raised by other reviewers. Therefore, I stick to my original rating of borderline weak acceptance. I appreciate the decision to revise the paper title to better clarify the scope; the new title will be more scientific than the previous overly-hyped one. I also value the new experiments comparing models using their XAI method, which show that current models are biased towards color over shape and size, and that CNNs and ViTs excel in local and global pattern recognition, respectively. While these insights aren't entirely new, it's reassuring to see them align with prior work.
Great Appreciation to Reviewer
Dear Reviewer FVKA, Truly thank you for the nice words and the recognition of our efforts. We will incorporate all the suggestions to make our work more scientific. One more thing to double-check is that the author mentioned "stick to my original rating of weak acceptance". But the current rating is baseline acceptance rather than WA(6). Not sure if there is any misunderstanding from our side, any error in the system or R-FVKA just meant it. Anyway, your suggestions are valuable for us. Best!
Summary
The goal of the paper is to bridge the gap between human comprehension and AI decision. For this purpose, the authors propose a Language Model as Visual Explainer (LVX), an approach to interpret the internal workings of vision models through tree-structured linguistic explanations, without the need for additional model training. The propose method leverages the collaboration between vision models and large language models (LLM) to generate these explanations. The LLM is used to outline hierarchical visual attributes, while a text-to-image API retrieves images that best match these textual descriptions. By mapping these texts and images to the vision model’s embedding space, they create a hierarchical visual embedding tree. This tree is dynamically adjusted by querying the LLM with language templates, allowing for the addition new attributes and removal of irrelevant concepts based on the model’s representations. Additionally, the authors propose a new benchmark and new metrics to demonstrate the plausibility, faithfulness, and stability of the newly introduced method.
Strengths
Here are the paper's strengths: - it introduces a novel exaplainability method leveraged by LLM - it introduces a new benchmark and novel metrics for an efficient evaluation - the paper is well documented and clearly written. The contributions and objectives are clearly stated. - the review of the state of the art is comprehensive and covers most of the relevant works - the experimental validation is extensive
Weaknesses
Weaknesses: - some concepts presented in the paper require more details
Questions
Here are my concerns: - Eq. 3: How is the loss function $L_{HMC}$ defined - Section 4.1 -> Evaluation Metrics -> Plausability: How is the unnormalized TK score defined? - How does your approach may cope with fine-grained datasets, such as Flowers or CUB-200? I assume that the robustness of the approach relies on the capability of the LLM to cope with fine-grained datasets. Do you think that your approach could handle such case or some modifications (fine-tuning) should be performed on the LLM in order to deal with this particular case. Please provide your insight with respect to this problem.
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
Limitations are related with the curation of the new dataset and imply the following aspects: false positive simages, the presence of out-of-distribution samples and potential bias in the dataset. This research work does not have any negative societal impact.
Summary
The paper suggests a new approach to building a hierarchical tree of visual attributes (represented with language) that matches the decision-making mechanisms of classifiers. The approach is based on using an LLM to suggest a tree of attributes that are related to a specific class, and then refine the tree according to its matches to the training date (this is done by comparing image embeddings of per attribute synthetic generated images and the train images). The hierarchical explanations are then evaluated using three measures (defined by the authors): plausibility, faithfulness, and stability. The tree explanations are shown to be beneficial when fine-tuning the classifier to match better the found structure, which slightly improves performance. The authors also exemplified the usage of the explanation for characterizing miss classification.
Strengths
* The idea of constructing a language description tree of a classifier is great! (although I'm afraid the method mainly visualizes the training images and not the network operations, see comments in weaknesses section) * The use of LLMs to generate and refine explanations inssures the generation of human-like descriptions, wich are human understandable. I liked the idea of refining the tree using the LLM.
Weaknesses
* My main concern is that unintentionally *the suggested tree representation mainly explains the training set, and not the network mechanism* (in a very complicated way): (1) Solely using the last layer of the net for the image embedding: Classifiers are known to have a hierarchical structure of attribute representations that are combined to make the classification decision. By looking only at the last layer embedding the author completely disregards that, and the tree representation constructed does not really explain the inner mechanism of the network but only the mechanism of one particular layer. (2) The refinement stage is made on the training samples; This means that relevant class attribution can be directly extracted by characterizing the leading attribution of the training data, with no connection to the network mechanism itself or any of it embeddings, without any need to synthesize synthetic data. This stage, combined with the fact that the image representations are extracted only from the last layer, makes the tree representation mainly visualize the hierarchy of the training set not the hierarchy of the operation of the network. * My second main concern is *the usability of this type of visual explanation*. The author provided some experiments for potential applications but these seem limited; Finetuning classifiers to better obey the representation only marginally improves performance compared to baselines (no baseline performance of the model before fine-tuning was provided). Using the tree for misclassification examination seems only anecdotal with no large-scale explanation or evaluation. Can the trees replace the classifiers and be used to classify images? Is it possible to compose the tree to get a model lever tree? * Does the explanation tree go beyond the structure of the hierarchy provided in context? I can imagine other important attributes, like the presence of singular objects vs many objects, or different types of conditions like the presence of a dog but only if it is leashed. According to the text, it seems like the “subjects” of each node are fixed according to the in-context example, which significantly limits the flexibility of the representation. Furthermore- how was the format in-context example decided? it seems to have a lot of influence on the results but that was not discussed in the paper. * In the tree refinement stage, there is no definition for what does it to have “nodes that are seldom visited”, this seems to be an important variable. what is the criterion for that? * Evaluation- The faithfulness score seems to directly connect with the way that the tree was defined. Baseline comparisons are rather ablation study. * The clarity of text and figures can be improved. Intro is very general and some definitions can be interpreted in many ways, for example, it will be much clearer if the authors state that by in vision model they mean a classifier. Another example is defining the image embedding way before using it which makes it hard to follow details. Also unclear how L_{HMC} is calculated. The scheme in Fig. 1 was unclear to me. fiuger 5- how were accuracy and MCS calculated here?
Questions
see weaknesses
Rating
4
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
limitations are discussed in the appendix
Thank the Reviewer for the Constructive Comments
Dear Reviewer UiCA, We would like to thank you again for your constructive comments and kind effort in reviewing our submission. Please do let us know if our response has addressed your concerns, and we are more than happy to address any further comments. Thanks!
Thank you for your detailed reply. I still wonder how the method is different than constructing the tree based on *only* examining the attributes of images of a given class C in the training set (without feed-forwarding them to the network). Can you please clarify that?
Response to Reviewer UiCA
Dear Reviewer UiCA, Thank you for your question. We are truly grateful for your time and the opportunity to clarify our approach. ------ If our understanding is correct, your main concern seems to be why our LVX needs LLM to refine trees, instead of using pre-existing attributes from the training set directly. To address this, we provide a brief response in `Q1` and an expanded explanation in `Q2`. Additionally, we illustrate the difference with an example in `Q3` and offer a quantitative comparison in `Q4`. `>>> Q1`**Quick Answer** `>>> R1`LVX **does not** merely replicate attributes seen in the training set. Instead, it dynamically adjusts attributes—sometimes *incorporating unseen attributes or excluding existing ones in the training set*—based on the classifier's perception. `>>> Q2`**Detailed Answer** `>>> R2`We can best clarify this by comparing two methods that focus solely on *examining training attributes* and explaining how they differ from LVX. **Method 1: Human-Annotated Attributes.** In this method, each class in the dataset comes with detailed, human-annotated attributes. There is no need for further refinement by an LLM; these attributes are organized into trees directly. We retrieve images for each attributes. For a test image, we match it to these generated image like a nearest-neighbor classifier to determine attributes. **Method 2: LLM-Generated Fixed Attributes.** Here, attributes are initially generated by an LLM based solely on the class name. This method represents the LVX in its initial stage, without refinements. Compared to LVX, those methods differ mainly in three ways: 1. **Attributes $\neq$ Explanations:** Unlike `Method 1` and `Method 2`, where attributes are predefined and not influenced by the classifiers, LVX refines these attributes to better explain how the model sees images. This makes LVX not just about predicting attributes but providing model-specific explanations. 2. **Static vs Dynamic:** `Method 1` and `Method 2` involve a finite set of static attributes. LVX, however, can add or remove attributes based on real-time classifier feedback. It enables open-vocabulary capabilities, as detailed in our paper `Line 108`. 3. **Annotation cost:** LVX eliminates the need for annotation required by `Method 1`, reducing costs. `>>> Q3`**Example: LVX Adds Attributes Not Found in Training Data** `>>> R3`We provide an example where LVX introduces attributes not originally present in the data. In `Fig 6` the top-right example, the `Hook` is mis-classified as `Nematode`. Actually, all images of `Nematode` in ImageNet are *black and white 2D microscope images* and do not have 3D attributes like **"Cylindrical"**. However, LVX introduces such non-existent attributes to the attribute tree, which helps explain why such misclassifications occur. This is not achievable for `Method 1`. `>>> Q4`**Quantitative Comparison** `>>> R4`We do an experiment to compare LVX with `Method 1` and `Method 2` in terms of faithfulness and plausibility for the generated trees. - **Setup**: Since ImageNet and CIFAR do not have human-annotated attributes, we use the CUB-200 dataset for our experiments. This dataset has 312 binary attributes organized into tree structures, which is ideal for `Method 1`. We compare LVX with `Method 1` (human-annotated attributes) and `Method 2` (LLM-generated attributes without refinement) using the ViT-S classifier. We measure faithfulness using the MSCD score and plausibility using the MCS score. - **Results**: The table below shows the performance differences. `Method 1` produces trees that align more with human recognition, indicated by a higher MCS score. However, it does not capture the classifier's internal workings as effectively as LVX, which *achieves a significantly lower faithfulness MSCD score*. `Method 2` performs the worst on both metrics. | Network | CUB-200 (MCS↑/MSCD↓) | | | |--|--|--|--| | |LVX | `Method 1`| `Method 2`| | ViT-S|28.64/**-1.592** | **29.13**/-0.532 | 26.32/-0.478| ------ Once again, thank the reviewer for the thoughtful feedback and for recognizing the potential in our work. We will incorporate the discussion in our revision. Please let me know if you need any further clarification.
Thank you for the additional explanation. I strongly suggest that all these additional explanations will appear in the paper. The paper and figures still need some major revisions to improve readability. I decided to increase my score to 4.
We truly thank Reviewer UiCA for raising the score and all the supportive feedbacks. We plan to include all analytic and ablation experiments in the paper and revise the writing to further strengthen our arguments. Here’s a quick plan for our revisions: `>>> Q5`**Revision Plan** `>>> R5` 1. For the *experimental part*, - We'll discuss why using training set attributes doesn't improve explanations (`R1-4`); - We'll show how removing the training set images worsens results (`A3`) ; - We’ll add more analysis and insights using LVX to explain the models (`A4`); We hope to include these updates in the main paper, as the NeurIPS allow one more page for camera-ready submission. 2. For the *writing and figure part*, we realize that the current content is overly dense—lots of information and long descriptions have made some figures too small. As suggested, we'll streamline these sections, simplify the logic, and increase the font size for figures for better clarity. ----------- Should you have any further questions or concerns, please let us know and we will try our best to clarify. Again, big thanks for all the suggestions—they really help us and this paper a lot!
Summary
This work proposes a method to understand the prediction of an image classification model using a tree of attributes. The tree of attributes is originally constructed in a text-only by querying gpt for identifying attributes of given concepts. They then use an image-to-test model (or retrieval model) to obtain a set of images corresponding to each attribute. At inference time, they match the input image to each attribute at the current tree level. The path from the root node to the leaf node is meant to interpret the models decision making process.
Strengths
This method works on models that are not just VLM models. Similar works rely on models being open vocabulary so that the image can be compared to arbitrary attributes described in text. This method first collects the attribute in text and then converts them to image embeddings so that models can be evaluated even if their image embedding space is not aligned with a text embedding space. The method is clearly described.
Weaknesses
The primary weakness of this work is that the way the tree is constructed is not affected by the image classification model at all but the explanation for the inner working of this model must be a path that exists in the tree. This then presupposes that the tree contains the same reasoning path used by the image classification model which seems to be a large assumption. For example, the model may identify a certain kind of bird by some spurious correlation in the training data. However, as the tree is created by a language model which does not know about this data, this is very unlikely to exist in the tree. However, this method will output some sort of path which is meant to be the explanation even though it is impossible for the tree to produce the correct explanation in this case. The evaluation metric for measuring the accuracy of the explanation seems poorly defined (see questions sections). Human eval doesn’t make sense in this case as humans cant evaluate how well a given explanation corresponds to the true decision making process of the model. (they can only measure how well it corresponds to a process they may have used) Some motivation for this method seems unfounded. For example, the authors state that “Upon observing a dog, we naturally check for the visibility of its tail.” I think this is not correct.
Questions
Faithfulness Score: Can you explain more directly how this score measures the inner workings of the model? Figure 3 uses the wrong form of “dear”/“deer”
Rating
5
Confidence
5
Soundness
2
Presentation
4
Contribution
2
Limitations
yes
Thank the Reviewer for the Constructive Comments
Dear Reviewer U6GY, We would like to thank you again for your constructive comments and kind effort in reviewing our submission. Please do let us know if our response has addressed your concerns, and we are more than happy to address any further comments. Thanks!
Thank you to the authors for pointing out that the tree is able to be influenced by the Image Models during the refinement process. I also appreciate the new experiments which demonstrate that this refinement process does lead to a better score. I also thank the authors for adding citations to support their biological inspiration. I do however still hold two of my concerns from the previous round of reviews: 1. Even with the influence of the Vision model during the refinement process, the tree can only be updated by the LLM. This means that the LLM must generate at some point the true reasoning behind the visions models decisions. I think it is not always reasonable to assume that the LLM will do this as the LLM is likely trained on human understanding of concepts which may not always align with the internal working of the model. 2. I still do not believe the faithfulness score truly measured the inner workings of the model. Just because the models internal representation of an image is close to the prototype of a given attribute, does not mean that the model is classifying an image a certain way because of that attribute. I will raise my score to a 4, as the authors addressed a portion of my concerns. However, I believe this work needs more investigation into if the evaluation metric is actually measuring the models reasoning.
Thank the Feedback from Reviewer U6GY
Dear Reviewer U6GY, We are sincerely thank you for raising the score and your time to review our work. Besides, we still want to clarify some of the points the reviewer raised. `>>> Q1`**LLM must Generate at some point the True Reasoning** `>>> R1`Yes, you are right, we somehow assume LLM can eventually generate true reasoning, even if not in the first round. While this assumption may have limitations, it offers benefits in terms of **human interpretability** and **efficiency**, as shown in `Q3`. We have also attempted to address potential issues through *tree refinement*, which optimizes the reasoning paths that should be explored. In addition, this assumption—that explanations can be generated by a model—is common. For example, it is used in counterfactual explanations with diffusion models [A, B] and in training LLMs to generate explanations [C]. We acknowledge that this approach has its limitations and will be explored further in our ongoing research. **References:** - [A] Jeanneret, G., Simon, L., & Jurie, F. Diffusion models for counterfactual explanations. ACCV 2022. - [B] Prabhu, Viraj, et al. Lance: Stress-testing visual models by generating language-guided counterfactual images. NeurIPS 2023. - [C] Hernandez, Evan, et al. Natural language descriptions of deep visual features. ICLR 2021. `>>> Q2`**Is the Faithfulness Score Really Faithful?** `>>> R2`Thanks for the nice question. We believe that measuring feature distance inherently provides a faithful explanation of the model's behavior. This is rooted in how classifiers are trained: Deep classifiers are *trained to perform prototype matching*. - **Mathematical Rationale:** Consider a feature extractor $g$ with a linear classifier $W$. Deep networks are trained to minimize the classification error $\text{min}_{g, W} L(W \circ g(x), y)$. If we view each column of $W$ as a prototype $W_i$, this objective translates into a prototype matching loss, aligning the sample's embedding $g(x)$ with the prototype $W_y$ of its class $y$: $\text{min}_{g, W} L(W_y, g(x))$ In this way, classifier make prediction based on its feature distance to its prototypes. In return, measuring this distance provide a faithful way to do explanation. - **Common Practice:** Despite above reason, using feature distance as a measure in prototype-based explanation is a standard practice in literatures [9,10] . We understand there are other ways to measure the causality or relevance between model decision and attributes. Using feature distance with prototypes is one of them and, of course, not a very bad one. ------- Once again, thank the reviewer for the thoughtful feedback and for recognizing the potential of our work.
I still have some concerns over this faithfulness evaluation metric as it does not seem to me to truly match the process of a classifier. For example, it cannot handle a case where a class either has both attribute A and B or neither (but never only one of these attributes). Essentially, I don't think a mathematical argument for a linear classifier on top of a set of attribute applies to this case (of a non linear classifier on top of a complex uninterpretable set of features). That being said, the authors make a compelling case over the use of this interpretability method in previous works. For this reason I raise my score to a 5.
Thank Reviewer U6GY for the question
Big thanks to Reviewer U6GY for raising the score and acknowledge our efforts! We appreciate your concern and fully understand that our faithfulness score isn't perfect; it involves certain simplifications, particularly in linear measurements over prototype similarity. It matches the classifier’s decision process, but up to the last layer. Luckily, despite these simplifications, this score can handle the co-occurrent attributes, in the case you raised. `>>> Q3`**Faithfulness score and Co-occurrence of Attributes** `>>> A3`That's a great question. In fact, our faithfulness score can handle the co-occurring attributes. It calculates the sum of feature distances, which can be explained as computing the *negative log-likelihood (NLL) of attributes A and B jointly existing* in the image. - **Probabilistic Rational:** Consider each attribute as an isotropic Gaussian $P(x|i) \sim \mathcal{N}(\mathbf{p}_i, \mathbf{I})$, with its mean at a prototype embedding $\mathbf{p}_i$. When assuming conditional independence among these Gaussians, the joint probability $P(x|1,\dots,N) = \prod_i^N P(x|i)$. By calculating the distance between an embedding $\mathbf{q}$ and these prototypes $\sum_i ||\mathbf{p}_i-\mathbf{q}||^2$, we are actually computing the negative log-likelihood that $\mathbf{q}$ belongs to the joint distribution. This can be expressed as: $$\sum_{i}^N ||\mathbf{p}_i-\mathbf{q}||^2 \propto \sum_i^N -\log(P(\mathbf{q}|i)) = -\log(P(\mathbf{q}|1,\dots,N))$$ Thus, the feature distance reflects the NLL, showing *how likely the sample contains all the attributes simultaneously*. While these assumptions may somewhat be overly strict, in the probabilistic view, our faithfulness score indeed can measure the co-occurrence of attributes. - **Joint Attribute Data:** Besides the probabilistic rationale, we also deliberately *collect data containing all attributes within a path from root to leaf*. For example, if the support image contains `a furry tail` for a dog, it will, of course, include `a tail`, because it is an ancestor node in the tree. So, given a test image, if multiple attributes on the same path are identified, they must co-exist in the image. In both sense, our faithfulness metric becomes a good indicator for co-occurring attributes. -------------- Should you have any further questions or concerns, please let us know and we will try our best to clarify. Again, thank your for all the suggestions and questions this week—they've really helped us to improve our paper!
Acknowledgement of Rebuttal
I want to thank the authors for addressing all my concerns.
Thank the reviewer again for your kind and encouraging comments! Best!
Decision
Accept (poster)