Scale Alone Does not Improve Mechanistic Interpretability in Vision Models

Summary

This paper presents an empirical evaluation of how interpretable the individual neurons in different models are, with specific focus on how increasing model scale effects this. This is done via large scale human experiments via amazon mechanical turk, and they find no evidence that larger models would be more interpretable, instead having weak evidence towards large models being less interpretable. They also release their evaluation data to help create automatic evaluation in the future via models to predict interpretability to a human.

Strengths

- Sound experimental setup and statistical analysis - Studying an important problem - Larger scale than previous experiments, finding additional evidence towards previous findings such as feature visualizations being less informative than natural images, and robust models having more informative feature visualizations - Clearly written - Good discussion of related work

Weaknesses

I think the experimental setup could be improved in a few ways: - I think 84 neurons per model, while bigger than previous might not be enough, especially if wanting to compare between layers in a single model, as this leaves less than 2 neurons per layer for ResNet-50 for example, which is not large enough sample size. This might also be part of the reason why most findings lack statistical significance. - While the costs make it harder to scale, could have been better to narrow down focus i.e. look at fewer models/layers within these models or use fewer evaluators per neuron. - Top-5% and top-15% activations for medium and large are too big of a jump in my opinion, top-15% rarely has any meaningful pattern for a neuron. Would be more interesting to see closer results like in the top-1% range Finally results are not surprising, I wasn't expecting scale alone or the other features studied here to have a significant effect on the interpretability. This reduces the impact of the work but it is still good information to have confirmed.

Questions

- How many trials were conducted? The main paper states more than 100k, but appendix A.3 only states 40k, where is the difference? - Why was the alternative image chosen from least activating images? Why not for example use a randomly chosen image as the other option? Or perturbed versions of highly activating images as done by Zimmermann et al.?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes, good discussion.

Summary

In this paper, the author performed a large-scale psychophysical experiment to see whether scales improve mechanistic interpretability. They found that there is no scaling effect for interpretability; the latest large models do not provide better mechanistic interpretability than older models. The dataset with user response is publicized so that it can be used in future studies that aim to optimize interpretability. I think the paper is novel in that it answers a new question that was not explored in this way. However, since it lacks new theoretical results, substantial methods, or findings, I believe the NeurIPS datasets and benchmarks track would be a more suitable venue for the paper.

Strengths

1. The approach to answering the question of whether scales affect mechanistic interpretability via large-scale psychological experiments is novel. 2. The experiment was performed on various axes such as model sizes, dataset sizes, and model architectures. 3. The large-scale psychological experiment dataset would be a valuable resource for further research 4. analyzed the dataset using multiple ways. 5. The paper was well written.

Weaknesses

1. The paper does not provide substantially new methods or findings.

Questions

1. Is the hypothesis that scaling improves mechanistic interpretability a common perception?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. I am not sure whether the hypothesis that scaling improves mechanistic interpretability is a common perception. If not, the utility of the dataset would be limited.

Summary

This paper starts with the motivation of a recent growing field of mechanistic interpretability, in which the goal is to attempt to reverse engineering a given neural network (or possibly any function). The main hypothesis is that since recent models have grown in size in terms of number of samples and number of parameters, if models perform more human-like task, the extracted features might become more interpretable. The authors then perform a large-scale human psycho/meta-physical study, in which they ask participants which of the two presented group of images are more aligned with a selected neuron’s response. The correctness of the response then serves as a metric for how interpretable a neural network is, in which they find neural networks from recent years (2017-2023), despite being larger in parameters and datasets size, are not mechanistically interpretable, which means study subjects are unable to identify the correct group of images. Last but not least, the authors release the data collected from participants as a dataset, to motivate the need for an automated formula/system to promote interpretability within any trained model.

Strengths

Overall, the paper is well-written and its contents are well-organized. I did not run into trouble understanding the paper. Interestingly, the work is heavily influenced by Olah 2017, which looks into activation maps of individual neurons of a neural network and find “object detectors” such as “dog neurons.” It is also somewhat parallel to works done back in 1950s, when neuroscientists would probe neurons in monkey visual cortex and find “edge” detectors from simple/complex cells. Hence I believe this experiment, although somewhat similar to what previous has done in terms of methodology, is an important contribution to the field of mechanistic interpretability, and to the entire field of interpretability/study of neural networks on a unit level in general. Although the connection to neuroscience in mentioned in the related work, personally I would suggest adding a short sentence to the introduction regarding its parallel to neuroscience. The setup of the experiment is also reasonable and sound. Authors have sufficiently considered the potential biases of the experiment and have addressed them accordingly in the design of the experiment. The experiment is also done with a wide range of models (both CNNs and ViTs) and datasets of different scales. I believe the experiments are well thought out and done.

Weaknesses

In this work, the authors constructed the hypothesis “If models make more human-like decisions, this might hint at a closer alignment between the extracted features and human perception.” While ImageNet classification is often considered standard for benchmarking, it is hard to argue that better models of predictive performance/larger models implies more human-like decisions. A better benchmark would have been comparing models that can perform multiple tasks, which humans do. An alternative solution would be to perform the same experiment on architectures, but that would obviously induce a much higher financial cost. I think the authors can better address why ImageNet is an indicative task for human-like decision in their Method or Introduction. Small typo. In Training Details on OpenReview website, authors wrote: “we do not train but onyl evaluate models”, onyl -> only But other than the above, there are no major weakness in this work.

Questions

1. Have the authors considered performing a similar experiment but with language tasks, since language is a more natural form of interpretability and method of showing one’s understanding? For instance, finding nearest neighbors of activation maps (as vectors), and ask whether the neighbors form any semantic clusters. 2. The definition of mechanistic interpretability stated by the authors: “understanding the internal information processing of deep neural networks by attempting to reverse engineer them.” It is not immediately clear to me what it means when the authors stated in the conclusion that we need to optimize for this property. Can the authors elaborate on that? Does it imply we should design neural networks differently? Does it mean we should change the learning objectives we use to train the networks? Do the authors mean that we should design models with the ability to reverse engineer them? But wouldn’t that imply we already know how neural network works?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations in their Limitation section.

Summary

The authors investigate whether features associated with neurons in vision models become more or less interpretable as these models are scaled up. They utilize mechanical turk and a forced association test to see whether human workers are able to clearly associate exemplar images (or synthetic feature activations) to strongly vs. weakly activating sets of input images for given neurons. They find that across multiple scales and classes of vision models, there is no significant increase in interpretability when increasing scale. They provide a number of additional findings on the shortcomings of feature visualizations, and trends (or lack thereof) in interpretability as a function of neuron specificity and the location of neurons within a network. In addition to these interesting findings, they release an annotated dataset of neurons and their associated intepretabilities which should prove a great boon to mechanistic interpretability researchers more generally.

Strengths

Overall this paper provides a meaningful contribution to the field of interpretability research; both through its own investigations and through the accompanying release of the IMI Dataset. ## Originality and Significance The psychoanalytic approach toward addressing questions about interpretability has thus far been relatively underexplored, and is used to great effect in this work. The findings themselves seem general enough to guide future research on interpretability methods for vision models, but the release of the associated dataset may be more impactful still, as it will allow researchers with fewer resources to work on superior algorithmic approaches for discovering features / Interpretable neurons. The additional experiments carried out in App. B.4 are also very interesting, as these undermine some intuitions prevalent in the community regarding which kinds of simple flags are likely to indicate interpretable neurons (though the annotations don’t explicitly measure the levels of polysemanticity, thus allowing this to remain as a potential confounder). ## Quality and Clarity The methodology and findings are mostly well-presented, and only minor changes are suggested (see Weaknesses)

Weaknesses

The methodological approach seems comprehensive and well-suited towards addressing the questions of interest. I have no major criticisms. The two minor criticisms are on: * **Claims** - the paper repeatedly makes claims of the form “mechanistic interpretability of larger models is not better”, which is supposed to be short-hand for “the interpretability of features associated with neurons in larger vision models is not better”. I think this amounts to an overstatement of the valid claim without being any more concise, as reframing “feature interpretability” to “mechanistic interpretability” is a generalization (there are other forms of mechanistic interpretability which may be unaffected by these findings, such as circuit discovery). Collapsing “larger vision models” to “larger models” is fine, given that this is obvious. Additionally, one might wish to argue that “feature interpretability” is too broad, but the field’s current definition of features is quite loose, and the paper acknowledges issues around superposition and synthetic feature generation, so this is fine. * **Presentation** - The paper is very well presented for the most part. The only part that was slightly difficult to parse was Fig.2 - here I found the examples in the appendices more illuminating (partly due to a nitpick regarding the caption, mentioned below). I don’t think there would be any loss of generality if this figure were replaced with one of these examples from the appendix (as the caption explicitly states that the query may be a natural exemplar or synthetic visualization) ## Nitpicks: * 74: -> “was their usefulness .. experimentally quantified” * 114: do you mean “higher density of interpretable units”? * Fig 2 Caption: It was slightly unclear at first that “two extremely activating” meant one extremely positively, and one extremely negatively activating; this made the following “pick the more positively activating query” slightly confusing. * Fig 3 Caption: The caption is split into part “A” and “B”, but these are not labelled pictographically. Changing to “left” and “right” would be easiest. * 147 - “keep” -> “kept”, as latter part of sentence is past tense * 174 - “paradigm shift” seems quite strong * 300 - The footnote referencing superposition should be places before the period

Questions

It would be interesting to know how the effectiveness of synthetic feature visualizations varies for different models, as there is some discussion of the difficulties involved in using existing methods to generate features for the different models. Figure 12 mostly answers this as one can infer the “difference between synthetic and natural interpretability for each model” but it is not directly shown. It might be nice to add a figure with e.g. “3 non-cherry picked examples of synthetic feature visualizations” for each model, alongside the difference in synthetic vs natural interpretability. Perhaps this is not interesting as the feature visualizations are generally quite poor (as per Sec. 4.2).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations are thoroughly discussed.