Evaluating Neuron Interpretation Methods of NLP Models

Summary

This work investigates interpretation methods in NLP that identify which neurons in a neural network are most related to particular concepts (e.g. a specific part of speech). The key idea is to compare how consistent one method's ranking of neurons is (w.r.t. a specific concept) with that of all other considered methods. The assumption is that the method that produces the most consistent rankings is the one that is closest to the true ranking of important neurons. The authors consider a number of interpretation methods and pretrained language models and show that across several settings, the method that is most consistent is Probeless (Antverg and Belinkov, 2021).

Strengths

- The work considers the problem of interpreting the neuron-level representations of neural networks, which seems to be a recently emerging subarea of interpretability within NLP. I'm not fully convinced by its utility but there are indeed several recent works in this area so perhaps there would be interest in this work - The paper is mostly well written and motivated

Weaknesses

The main weakness is the assumption that consensus with existing interpretability methods is desirable. This consensus of course depends on the other considered interpretability methods, and it's not clear that the number of total interpretability methods (6) is sufficient to lead to reliable results that are based on consensus.

Questions

How do the insights about which interpretability methods is the most consistent change according to the considered interpretability methods? For example, can the authors remove 1,..n-1 interpretability methods from the consensus calculation and compare the results with those from the full set of n interpretability methods? Minor questions / notes: L100: are the examples supposed to be country names? The provided examples are cities. There are a lot more "random"concepts in a corpus than target concepts. Are the random concepts that are used for evaluation sampled such that they are equal in number to the target concepts? How consistent is this evaluation over different random samples? L185: why does the performance of a probe using random 100 neurons being higher than other neurons mean that the probe is memorizing? As the authors point out earlier in the manuscript, the concept knowledge is distributed so it makes sense that as one increases the number of random neurons that are being considered, the prediction performance would improve. If the authors were looking at randomly initialized models rather than pretrained models, then I would agree with their conclusion, but in the current setup, there seem to be alternate explanations for their results.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

See under Weaknesses

Summary

This work evaluates six different interpretation methods from a unified perspective. The authors focus on two challenges in this field: the absence of standard metrics and the lack of benchmarks. They propose two voting-based metrics to evaluate the compatibility among these six methods. Probeless consistently achieves the highest compatibility across all models based on their evaluation methodology. Further analytics experiments provide insight to the research community.

Strengths

1. The findings of the interpretation methods mentioned in this paper may foster research on the topic. 2. The idea of this paper is slightly novel.

Weaknesses

1. The selected six methods need to be more novel. More recent advancements in this field may exist rather than relying on L1 & L2 regularization. 2. The validation of the superiority of Probeless needs to be more comprehensive. 3. The mentioned problems (metrics and benchmarks) were not effectively addressed, making this method difficult to be reproduced in future applications.

Questions

1. Are there additional performance comparison experiments to practically demonstrate the effectiveness of this method? For example, try to deactivate the most important 10% of neurons calculated by each interpretation method, and then observe whether Probeless exhibits the most significant performance decline. 2. To assess the efficacy of an interpretation method across various fields, is it essential to replicate multiple interpretation methods within those fields before applying the voting metrics? If so, this process can be quite challenging and should be acknowledged as a limitation in this work.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is no potential negative societal impact.

Summary

This paper provides a comparative analysis of six neuron interpretation methods utilizing diverse concepts across three distinct pre-trained models and introduces an evaluation framework predicated on voting theory. Importantly, it offers the first comprehensive examination of multiple neuron interpretation methods and strives to mitigate the challenges in this field. The authors note similarities among the most proficient techniques within layers of neurons, with these resemblances remaining regardless of methodological deviations. It is thereby suggested, existing neuronal interpretation methodologies might have focused on a reciprocal group of top-performing neurons. The paper also discusses the lack of no recognised evaluative metrics along with the lack of gold annotations by suggesting an evaluation strategy that consists of two compatible metrics with a pairwise comparison. This approach could facilitate the creation of a means to evaluate new neuron interpretation methods.

Strengths

The major strength of the paper lies in its originality; being the first work to create a ranking metric using voting theory for comparing the various neuron interpretation methods available. This methos should allow for autumatic comparison between existing and new neuron actiavtion methos. Furthermore, it embarked on a comprehensive comparative analysis via this new evaluation framework

Weaknesses

The primary weakness of the paper lies in its disregard for significant, influential studies previously published in the domain, thus lacking a comprehensive deliberation on an extensive range of neuron activation methods studied by the mechanistic interpretability community. In addition, the paper grievously overlooks the complexities associated with polysemantic, superposition and neuroplastic neuron behaviors encountered in neuron activation. Work worth considering are: 1- Elhage, et al - https://transformer-circuits.pub/2021/framework/index.html 2 - Olsson et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html 3- Elhage, et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html 4- Henighan et al - https://transformer-circuits.pub/2023/toy-double-descent/index.html 5 - Foote et al - https://arxiv.org/pdf/2305.19911.pdf 6 - Bills et al - https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html 7 - wang et al - arXiv preprint arXiv:2211.00593 8 - Chan et al - https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing 9 - Tenney et al - https://arxiv.org/pdf/2008.05122.pdf

Questions

Do I understand it right that if the methods under study are sub-optimal, the voting results can also be sub-optimal? In other words, the voting method does not provide any indications of how well a method may perform. Rather, it simply allows for a comparison between them. Considering that we are in the early phases of understanding the black-box nature of ML models, is it appropriate to start discussing benchmarks now? I have my reservations due to our lack of comprehensive hypotheses regarding their inner mechanisms. Hence, it presents a dilemma because it's as if we might not be in the position to start comparing or benchmarking.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors provide some insightful limitations of thier work.

Summary

This paper proposes a standardized evaluation metric and benchmark for comparing various neuron interpretation methods, based on ideas from majority voting. The benchmark is based on the hypothesis that "neurons that are commonly discovered by different interpretation methods are more informative than others", and uses this hypothesis to rank & score methods by their alignment with the majority (in terms of the neurons they assign behaviors to). I discuss some thoughts/limitations of this hypothesis below (and the authors discuss it as well in their limitations section). The authors demonstrate empirically why the current commonly-used evaluation metric (fitting a classifier to the neurons to predict the concept) favors certain methods over others and suffers from the same common issues as probing classifiers in general. Then then propose two "compatibility"-based metrics, one set-based and one rank-based, for comparing the top-k neurons deemed informative for explaining a concept by each method, and provide in-depth analysis using the metrics to compare 6 different existing neuron interpretation methods (3 corpus-based, and 3 classifier-based). Overall, I believe this paper is sound, well-executed, thorough, and clear, and will serve a useful role in the neural model interpretability sub-community. Based on the authors' response to my below questions, I may be inclined to raise my score further. Edit: I read the authors' rebuttal and it addressed my concerns. I did not change my score as I originally insinuated above, in part due to finding validity in the other reviewers' critiques. I still think the paper should be accepted.

Strengths

Originality & Significance: - The paper serves an important role in the mechanistic interpretability sub-community, and is likely to be of good value to this community. - I am not aware of any other attempts to organize methods in this manner, or propose a standardized benchmark. Quality & Clarity: - The experimental section is sound. The lack of ground-truth in makes evaluating explanation evaluation metric "faithfulness" difficult, but the authors have nonetheless provided a convincing set of results. - The paper is well-organized and well-written. The paper uses proper mathematical notation throughout. - The authors have promised to release the code upon publication.

Weaknesses

Agreement with hypothesis underlying the benchmark: - I am not sure I am totally convinced by the claim that the most informative neurons will be discovered by the most methods, given that neurons that are easily discoverable can often be those which are perhaps more simplistic in what they encode (e.g., only serve a singular function rather than multiple, concept is encoded only in that singular neuron rather than spread over multiple, etc.) However, the granularity and assumptions underlying neuron interpretation methods is a nascent research question in the interpretability community overall, so I will not fault the authors for their hypothesis (i.e., I think the paper is appropriately scoped and the hypothesis is reasonable in the context of existing/popular neuron interpretation methods). It would be nice for the authors to discuss limitations of the hypothesis (or neuron interpretation methods in general) in the limitations section. More minor: - The paper is lacking a related works section. While the methods studied in the paper are described in detail in Section 2, it would be nice to have some sort of summary paragraph of the field of neuron-level interpretation as a whole after the introduction to situate the work (and potentially mention other methods that are not tested in the paper). - There is some minor conflation it seems between the decision to treat each neuron interpretation method as producing a ranked list vs. performing binary set membership classification. This is particularly confusing with the classifier methods, which are directly trained to do the latter (4.1) but tested on the former via the NeuronVote method. However, the inclusion of the set-based metric, AvgOverlap, mitigates this issue to some extent. I think it would be good for the authors to clarify early in the paper that each method *can* be viewed as producing a ranking, even though this is not always how the methods have been proposed or used in practice, and that the inclusion/comparison of *both* evaluation metrics is designed to provide a view that does not unnecessarily favor one method over the other, by considering both the set membership and the rank views.

Questions

- L12: colon instead of semicolon? - L20: ungrammatical sentence - Would be good to introduce the term "probe" somewhere in l.37-43 of the introduction (or earlier). You mention the "Probeless" classifier multiple times in the intro/abstract, but don't define this term anywhere. - L102: represents --> represent - L150: extra "i" - L205-206 describe AvgOverlap as set-based (not taking ranking into account); this contradicts lines 222-223, where I think the word "ranking" is meant as "set". Can you clarify? - I don't understand the role of Section 3.3 and its associated equation. Why wouldn't you just use Eqns. 7 and 8 with $|\mathcal{M}| = 2$? - L255: the choice of hyperparameter values here seems like it could have a big impact on the results; did you do any search? - Table 2 is never referred to in the text

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes- see one suggestion above.

Official comment by Reviewer QcFR

I thank the authors for their responses. While I agree that Bills et al. did not appear until May 2023, Foote et al. was available on OpenReview at an ICLR workshop and Arxiv in April 2023. The suggested missing references do not encapsulate a broader view of interpretability but rather those closest to the approach you considered in your paper. I would encourage the authors to examine recent developments in neuron interpretation methods that address the issue of superposition. For example: Sharkey et al.: https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition and Sharkey, Lee: https://arxiv.org/pdf/2305.03452.pdf To strengthen the limitations section, I believe a more thorough discussion of the challenge of superposition is needed. I am unsatisfied with the answer in the last response regarding "...appropriate time to start discussing benchmarks...". I believe the works considered in this paper lie under the subfield of mechanistic interpretability and not in the broader interpretability field. If the authors believe otherwise, I would be interested in seeing a paragraph in the paper discussing how methods considered in this paper do not lie under mechanistic interpretability. Without such a discussion, I am less confident in recommending acceptance.

Response to Rebuttal

Thank you for the response. I appreciate the addition of the related works section. I will keep my score.