CoSy: Evaluating Textual Explanations of Neurons

Summary

The authors propose a framework for evaluating Neuron Annotation methods which label a neuron in a given vision model, via a textual description. This framework is based on generating a set of images using a text2image model, given the predicted textual description of the neuron which acts as the textual input prompt to the generative model. The idea is that if the textual description truly describes the neuron, then the generated image should activate that neuron more than that of a random baseline image from the training dataset the model was trained on (denoted as the control image). The authors use two scoring functions: the AUC and MAD for quantitative evaluation. The authors further evaluate the effectiveness of the evaluation framework, showing that it is a valid way. They then proceed to evaluating existing Neuron Annotation works, and drawing findings and conclusions, such as the fact that some methods are generating random textual descriptions of neurons.

Strengths

I would like to congratulate the authors for their work. I really enjoyed reading the paper. - This work is the first to propose an unified, intuitive and valid way of evaluating neuron annotation works. Indeed, most of these works provide their own evaluation metrics which makes it hard to compare with previous works, and verify if one method is better than the other - The paper is very well written, understandable and straight-forward. The evaluation framework is also simple, something which all people and researchers will appreciate. - The papers provides a meta-evaluation section to show the validity of the proposed framework, which I find interesting and important - The finding that some Neuron Annotation works provide wrong predictions of textual descriptions (or let's say, statistical-based descriptions) is very interesting and important, which raises a concern in the interpretability field, especially in safety-critical applications.

Weaknesses

I could not really find major weaknesses that are grounds for rejection. There are some moderate weaknesses: - [W1] How does the proposed evaluation framework relate to the other evaluation methods used in existing Neuron Annotation works? Do they align well with the new proposed measure? For example, does the proposed evaluation framework align well with the BERTScores for the human annotations of MILAN? In CLIP-Dissect, they labeled the neurons of the final classification layer and measured accuracy with the ground-truth class names. Does the proposed evaluation framework align well here also? - [W2] The pool of models is a bit small. More models (especially for for ImageNet) should be analysed. ImageNet ResNet50 is a common model in most Neuron Annotation works and should be reported. What about self-supervised models, such as DINO? How do they compare to classifiers? MILAN also performs analysis on these models. - [W3] There is another related Neuron Annotation work [R1], which is only applicable to ViTs. Here, attention heads and tokens are considered as Neurons and labeled. The authors may extend the analysis to this work to strengthen the paper, given that [R1] is a mathematically-valid decomposition of the ViT, and therefore should align best with the proposed evaluation framework compared to other Neuron Annotation works using the ViT. - [W4] The Area Under the Receiver Operating Characteristic (AUC) in section 3.2 seems to be a measure which does not involve generating a curve. But usually AUC means that we have a curve in the first place before taking the area under it. What is the curve in the author's case? Other minor weaknesses: - The text2image models are very time-consuming (the authors also mention that it takes 12 min using a parallel GPU setup, for generating 50 images). This limits the applicability to new users that want to evaluate their works. It would be nice to report the speed of different Stable Diffusion models along in Section 4.1, and include other fast-optimized and less-computationally expensive models -- i am not an expert in the text2image field so i do not know which models in the current literature are faster than the ones the authors use. Fortunately, given how fast the generative text2image field is moving, this problem will not be a problem in the very near future. - The fact that CLIP-dissect achieves good results in not surprising - because CLIP-Dissect interprets CLIP Neurons rather than CNN/ViT neurons. Their method is based on scaling CNN/ViT neuron activations using CLIP. Therefore, any image-text pair judged low by CLIP will also reduce the CNN/ViT neuron activation value for the corresponding image. In essence, one could think of CLIP-dissect as an amplified value of the CLIP image-text score (amplified by the CNN/ViT neuron activation value for the same image). And since CLIP itself is a very strong model and especially in scoring image-text inputs, it is not surprising that is performs best. - Line 156, "synthetic" should be replaced by "natural images" (as you already used "generated images" in line 155). Also, please be consistent with terminology. [R1] INTERPRETING CLIP’S IMAGE REPRESENTATION VIA TEXT-BASED DECOMPOSITION, ICLR 2024

Questions

In general, i feel this paper is a clear accept to me, and it will have a good impact in the field. I would like that the moderate weaknesses are addressed. While I understand that Rebutal time is limited, and computation time of the proposed framework is also time-consuming, I encourage the authors to address as much as they can from the moderate weaknesses (W1-W2-W3-W4). The minor weaknesses are just comments that can be included in the final manuscript.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitations are discussed. No special issues with negative societal impact.

Summary

The authors present a new framework designed to evaluate the quality of textual explanations for neurons in deep neural networks (DNN). To this end, the paper introduces CoSY, which aims to provide a quantitative, architecture-agnostic evaluation method for these explanations, addressing the challenge of the lack of unified, general-purpose evaluation metrics in the field. Given a neuron $f$ of a DNN and explanation $s \in S$, CoSY evaluates the alignment between the explanation and a neuron in three simple steps. It utilizes a generative model to create synthetic data points from textual explanations and compares the neuron's response to these data points with its response to control data points. This comparison provides a quality estimate of the textual explanations. Further, they perform a large set of empirical analyses to quantify the effectiveness of their proposed AUC and MAD metrics.

Strengths

1. The paper introduces a new quantitative framework that allows comparisons of different explanation methods, facilitating a more standardized approach to evaluating the quality of neuron explanations​​. 2. The proposed framework is independent of specific neural network architectures, making it broadly applicable to various models in computer vision and potentially other domains. 3. The paper includes extensive meta-evaluation experiments to validate the reliability of CoSY.

Weaknesses

1. The authors indicate that "the results using prompt 5 as input to SDXL leads to the highest similarity to natural images" --- isn't there an implicit bias towards images with a single object in these results? Given that, most of the images in datasets like ImageNet consist of single objects, generating synthetic images using a concept and comparing them with natural images should only work for clean natural images, where the concept is explicitly observable. 2. The authors argue that "different methods devised their evaluation criteria, making it difficult to perform general-purpose, comprehensive cross-comparisons." While it is true that the community should work towards common benchmark metrics, it would be great if the authors could clarify what is the problem with the existing quantitative metrics. Further, why can't someone benchmark existing metrics? 3. In Lines 159-161, the authors mention that they employ cosine similarity (CS) to measure the similarity between synthetic images and natural images corresponding to the same concept, but they never discuss the problems or motivations of using CS. For instance, cosine similarity is not a well-calibrated metric (also observed in Fig 2 where we observe very little variance in cosine values across different prompts) for models like CLIP, where the representations are densely packed using a large number of concepts. 4. Can the authors comment how possible techniques to alleviate the dependency of a text-to-image generative model in their evaluation pipeline?

Questions

1. Should we expect the AUC and MAD evaluation metrics to correlate? i.e., will an explanation method achieving high AUC also obtain a high MAD score? If yes, why don't we observe this consistently in the empirical analysis? 2. Is quantifying the synthetic image reliability using only 10 random concepts from the 1,000 classes of ImageNet in Figure 2 justified? Do these results hold for a larger number of concepts? 3. Is the aggregation of the activation values done using absolute sum or signed sum of the activations? This is important as negative activations may bias the aggregate value and impact the AUC and MAD metrics. 4. The authors present a very interesting analysis in Fig 4, where they aim to study the quality of explanations for neurons in different layers of a model. However, the error in the values in Fig. 4 is very large, questioning the conclusions from the figure. Moreover, given that the first few layers of a model detect generic low-level features like corners, edges, angles, and colors, why don't we observe near-perfect cosine similarity?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

Please refer to the weakness section for more details.

Summary

The paper presents a novel, architecture-agnostic framework called COSY for quantitatively evaluating textual explanations of neurons in deep neural networks. The framework utilizes generative models to create synthetic images based on textual explanations, allowing for a standardized comparison of neuron responses to these synthetic images versus control images. Through a series of meta-evaluation experiments, the authors demonstrate the reliability and practical value of COSY by benchmarking various concept-based textual explanation methods for computer vision tasks, revealing significant variability in their quality.

Strengths

1. The problem of evaluating textual explanations of neurons is worth studying, which is critical for the advancement of explainable AI and the wider adoption of machine learning models. 2. The COSY framework is designed to be architecture-agnostic, meaning it can be applied to any computer vision model regardless of its underlying architecture. 3. COSY introduces a novel, quantitative evaluation framework for textual explanations of neurons, which addresses the lack of unified, general-purpose evaluation methods. 4. Through the COSY framework, various existing concept-based textual explanation methods can be benchmarked and compared. This is demonstrated in the paper by benchmarking multiple explanation methods, revealing significant differences in their quality and providing insights into their performance.

Weaknesses

1. The effectiveness of the COSY framework relies heavily on the quality of the generative models used to create synthetic images. If the generative models are not trained on a diverse set of concepts, they may fail to produce accurate synthetic images, thereby affecting the evaluation's reliability. 2. As mentioned by the authors, the INVERT method optimizes for the AUC metric during explanation generation; it may be biased towards achieving higher AUC scores in the COSY evaluation, potentially leading to an overestimation of its performance compared to other methods. 3. As noted in the paper, the COSY framework demonstrates a decline in explanation quality for neurons in the lower layers of the network, which typically encode lower-level features. Could the authors provide more insight into this phenomenon? Is it possible that this is due to limitations in the generative models' capabilities, given that lower layers generally encode more basic concepts which might not be well-represented by the generative model?

Questions

see the Weaknesses part

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations.

Summary

This paper proposes an automatic evaluation for textual explanations of neurons in vision models. The evaluation works by using a text-to-image model to generate images based on the explanation of a neuron. Then, these images are passed through the vision model and that neuron’s activations are recorded. These activations are compared to neuron activations on control images that should have nothing to do with the explanation. A good explanation should yield generated images that produce high neuron activations, as compared to control images. Two metrics are used to measure the difference in neuron activations. The paper validates the evaluation framework by checking that (1) generated images are similar to natural images of class concepts, (2) vision model neuron activations are similar for generated and natural images, and (3) that “ground-truth” explanations (class concept labels) receive high scores while “random” explanations (random class labels) receive low scores. Finally, the authors apply their evaluation to several prominent neuron explanation methods and draw some conclusions about which methods work better and at which layers.

Strengths

- Very important: This paper provides an automatic evaluation for textual explanations of neurons that should provide a reasonably reliable signal for explanation quality. Such an evaluation should serve to guide methods research in the area, and has been sorely needed. - Very important: The paper tackles an important problem, evaluation of explanation methods in computer vision. - Important: The paper is very clear, well-organized, and has good illustrations. - Important: Many small design choices in the paper are very reasonable, like the choice of metrics and prompt selection. - Important: The paper provides some empirical results with current explanation methods that highlight directions for future research in the area.

Weaknesses

- Very important: The meta-evaluation in this paper suffers a bit on two fronts. First, it is difficult to say it is a meta-evaluation in the sense that we could plug another evaluation procedure into this framework and mesaure how different evaluation results correlate with some *third,* more ground-truth or utility-driven evaluation of explanation methods. Take for example the meta-evaluation of automatic machine-translation metrics. Metrics like BLEU and ROUGE are accepted on the basis of their correlation (or lack thereof) with human judgment of translation quality. What would make the meta-evaluation in this paper more of a meta-evaluation is if it compared with, for instance, evaluations from prior work described in “Prior Methods for Evaluation”, using some third ground-truth/utility-oriented evaluation as a target measure (see e.g. evaluations in https://arxiv.org/pdf/2211.10154 as possible targets). To clarify why I don’t think the results in Sec. 4.3 count as such a ground-truth evaluation, my second point here is that there is a major distribution shift in what we want to measure between typical uses of textual neuron explanations and the validations conducted in Sec. 4. Specifically, Sec. 4 focuses exclusively on evaluations of “output neurons”, i.e. class logits, whereas textual neuron explainers are applied almost exclusively to intermediate hidden neurons in models. This means that even a proper meta-evaluation comparing CoSY against competing explanation methods would be limited by a narrow focus on output neurons and not intermediate neurons. I wonder, could the paper inclue a meta-evaluation experiment utilizing a model that has been trained with intermediate representation supervision, like a kind of concept-bottleneck model, so there could be some ground-truth label for an intermediate neuron (at least as ground-truth as using output neurons, which are learned with direct supervision)? - Important: I have some other doubts about how the evaluation will work for neurons that represent low-level (perhaps early layer neurons) or abstract features (possibly middle/later layer neurons). For instance, how about low-level edge/shape features, or abstract features like “white objects”. It seems guaranteed that the control images will share many low-level features with the generated images. And it seems very possible that the control images include white objects. These cases seem problematic in that they could lead to low AUC or MAD scores even if the explanation is correct. Can the control images be generated in a conditional way that encourages them to lack the concepts in the proposed explanations? - Important: Concepts could overlap or be nested in semantic hierarchies, and I am not sure that the evaluation framework could clearly discriminate between good and bad explanations in these settings. For example, we might want to distinguish between “red apples”, “red fruits”, and “red objects” as explanations for a neuron. In order to do this, we need the generated images for each explanation to achieve a sufficient amount of diversity. It would be a problem if when the text-to-image model gets the phrase “red fruits”, it generates mostly red apples (a very salient example of red fruit). So, I would suggest that one meta-evaluation experiment focus on measuring diversity or coverage of the generated images.

Questions

- There has been a shift in the literature from treating neurons as the unit of analysis to feature directions, starting with TCAV (https://arxiv.org/abs/1711.11279) and extending through work like CRAFT (https://arxiv.org/pdf/2211.10154). One example of automatic textual explanation of feature directions is given for LLMs in Sec. 3 of https://arxiv.org/pdf/2309.08600 (as well as https://transformer-circuits.pub/2023/monosemantic-features#global-analysis-interp-auto-acts). I don’t know if any existing methods for vision models could be adapted to focus on latent feature directions rather than neurons, but this would be a ripe direction for future methods. - Based on the above, the related work section might also point to (1) https://arxiv.org/pdf/2309.08600, (2) https://transformer-circuits.pub/2023/monosemantic-features#global-analysis-interp-auto-acts, and (3) https://arxiv.org/pdf/2211.10154, and possibly point to automatic, model-based evaluations of local explanations like https://arxiv.org/pdf/2312.12747. - I’m curious what you would think of fitting a logistic classifier to the neuron activations and computing that classifier’s AUC. How would that differ from your non-parametric AUC?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

I think the limitations discussion is sufficient.

Response to rebuttal

I thank the authors for the rebuttal and the extra experiments they have conducted. I am generally happy and positive about this paper, and I will retain my score as an "Accept". The authors should, however, incorporate these extra experiments, as well as other reviewers comments, in the revised manuscript. Regarding [A6], i was referring to the bias that CLIP-Dissect method carries towards CLIP (it explains CLIP rather than other models such as ResNet-50 due to the reasons I mentioned). I was not referring to the authors's method bias. This was not really a weakness, rather a justification of a finding of why CLIP-Dissect performs better. I wish the authors the best of luck and congratulations!

Response to rebuttal

Thanks for the thorough response! Some comments below: >Choice of wording:…Sanity Checks Thanks! Sanity Checks sounds good to me. >Comparison to prior evaluation: Thanks, I’m pretty sympathetic to the point that actually comparing against other evals fairly is difficult here. It seems acceptable, though not totally ideal, to judge how reasonable each evaluation framework seems on the merits. >Concept-bottleneck for lower-level representations: ...We believe the ground-truth nature of CB concepts and last-layer labels to be often similar. Specifically, mostly during training, intermediate concepts are learned in a supervised manner, often using a similar loss function as the output of the model [1]. Therefore we consider that there is little difference between CB neurons and output neurons. Well, a concept could be “fur” or “leg”. I think these concepts are much more low-level than the classes in ImageNet, which includes many specific animals. I agree that the loss functions may be similar. Overall, I think there is still some important distribution shift between intermediate (CB) concepts and class/label concepts. > [A2] Similarity of generated concepts to control data: Ok I agree with the claim about “white objects”. The control data should have few white objects, while the hypothesis data could have many, and MAD should reflect this. But I am still concerned about cases with other low level features, like edge detectors. I agree that with the claim about “inherent difficulty in describing low-level abstractions in natural language due to their complexity.“ It may be hard to describe an edge detector in words. But it would be nice to have evals that reflect this and penalize methods that fail to do this, if that is what a feature truly represents! The concern right now is that both the generated and control images will have a bunch of edges in them. So I think an even better eval would try to *remove* the the hypothesis feature from the control images. >[A3] Diversity of generated images or concept broadness…Appendix A.8 and Figure 9 I’m not sure I see A.8 and Fig 9. I see A.6 and Fig 8? Either way, based on A.6 and the rebuttal, I can accept that the generated images seem diverse enough for purposes of rank ordering explanations by their quality. It could be nice to show that the rank ordering of methods is similar across different diffusion model sampling temperatures (as well as other details of the diffusion model, of course), but I’m satisified with the results here. >Thus we considered only the canonical basis, to establish a fair comparison. Makes sense! >[Q3] Fitting a logistic classifier to the neuron activations: Right, so you could take the activations for the control images and activations for the generated images and try to classify them as control vs. generated with a logistic regression. So it’s a regression with one feature, which is the neuron activation. Then you can compute an AUROC for this logistic model. Contrast this with the AUC derived from a pairwise comparison of all possible (a,b) activation pairs across the two groups, which is non-parametric. I actually don’t know what the difference would be. Just an idea. --- Based on the above discussion, I continue to be happy with the paper and maintain my score of 7.